Going to a "prestigious" college - say, one of the Ivy League schools - has a huge positive network effect on your career, if you want to capitalize on it. It's valuable social proof, at least for the first decade after you graduate, and counts for at least as much as merit and hard work, however frustrating that may be for the hard-working and meritorious high school grad determined not to go to college. You are going to get to know people and walk through doors that you just couldn't if you never went. Yes, Mark Zuckerberg dropped out of Harvard, but he still went, and happened to make a critical set of connections there.