I believe there is a lot of hidden fight behind the scenes for Google to monopolize the web.
There are a lot of expectations from the public that Google maintains. Apart from delivering the best search results, they have to respect robots.txt, limit crawling frequency, deliver search results really fast, etc.
At the same time, lots of people want to game Google and spam search results so they can make money with ads. The competition is not fair at all - the websites can tell which traffic came from GoogleBot and craft legitimate responses while Google is not allowed to publicly crawl the website with a fake User Agent.
Many websites are concerned about their data being crawled (like Amazon, they certainly wouldn't be happy if all their price information is dumped as a database), and Google has to make sure that no robot can crawl too much of a website by automatically searching Google, and to that end they invented reCATPCHA.
It's not easy at all to build a search engine that behaves responsibly both to websites and to users. It's the ability to deal with all these matters really well that gives Google the monopoly power.
There are a lot of expectations from the public that Google maintains. Apart from delivering the best search results, they have to respect robots.txt, limit crawling frequency, deliver search results really fast, etc.
At the same time, lots of people want to game Google and spam search results so they can make money with ads. The competition is not fair at all - the websites can tell which traffic came from GoogleBot and craft legitimate responses while Google is not allowed to publicly crawl the website with a fake User Agent.
Many websites are concerned about their data being crawled (like Amazon, they certainly wouldn't be happy if all their price information is dumped as a database), and Google has to make sure that no robot can crawl too much of a website by automatically searching Google, and to that end they invented reCATPCHA.
It's not easy at all to build a search engine that behaves responsibly both to websites and to users. It's the ability to deal with all these matters really well that gives Google the monopoly power.