Defence Secretary Gavin Williamson has refused to say if International Development Secretary Priti Patel should be sacked.