Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

If you try to mass scrape almost any major site (millions of pages of content) they'll block you.

For example, if you went one by one through Stack Overflow and sucked out every question and answer, your scraper bot would get banned (unless you're doing one request per minute, in which case you'll never finish).

Or if you tried to scrape Twitter.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: