Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Which large website did you get banned from for scraping? I did some scrapping and never got banned... perhaps my scraping's rate was not too fast.


If you try to mass scrape almost any major site (millions of pages of content) they'll block you.

For example, if you went one by one through Stack Overflow and sucked out every question and answer, your scraper bot would get banned (unless you're doing one request per minute, in which case you'll never finish).

Or if you tried to scrape Twitter.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: