Greg Clarke has admitted he only spoke to Bury owner Steve Dale after the club had been expelled from the Football League.