> You have to explain yourself and there is the implication that wearing a suit is somehow inappropriate.

If you're the only one wearing a suit, you look overdressed. It's as simple as that.