I empathise with your thoughts on this. It occurs to me that the different words aren't consistent across cultures and sometimes the word man and woman don't feel equivalent. Sadly I didn't pay attention when being taught grammar so can't explain why I think that in detail.

What do you think of the word lady? Gentlemen doesn't seem to be the male equivalent anymore.