Lord Heseltine says he wouldn't "sacrifice" his own judgement for the sake of Britain's "self interest".