Tory MP and former Attorney General Dominic Grieve says talks with the Government have turned into a "dialogue of the deaf".