Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Yeah, MSNbot isn't very friendly, so they somewhat bring it on themselves. Both it and Yahoo's bot are much worse than Google at predicting likely page updates, as well--- on one of my sites that has a blank robots.txt, MSNbot accounts for >10x as many hits as GoogleBot, yet Google still manages to keep its index just as up to date.

Some people have had success with crawl-delay, though there are occasional reports of MSNbot not honoring that either.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: