The security operation targeting Rohingya Muslims in Myanmar "seems a textbook example of ethnic cleansing", the UN human rights chief says.