Context is important, but your point that the phrase can be easily misinterpreted is well taken.

The phrase should certainly not be used if someone merely complains about a problem that men have. Rather, it is intended to be used when someone tries to derail a discussion about prejudice against women by talking about how hard it is for men.

Note that the comment I was replying to was in a thread about education, and it immediately started trying to shift the discussion to suicide rates and the judicial system.