Conservative councillor Matthew Palmer was filmed mouthing "do not let them in" as protesters tried to enter a meeting.