A committee of MPs is warning that ministers risk giving the impression that the issue is "too trivial to address".