Athletes aren't exactly a random sample. They are people who have self-selected to spend time on athletics.

I also don't think the stereotype is accurate. We tend to confuse interest in pursuits that are perceived as intellectual with actual intelligence.