James Cleverly, the new deputy chairman of the Conservative Party, has denied claims that diversity influenced the reshuffle.