International Development Secretary Liam Fox says the Brexit referendum was legitimate and people should stop fighting over it.