Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Sorry to spoil the fun, but I've disallowed /view/ from robots.txt to prevent accidental shares. I think this one must have been from someone sharing on Twitter or some other public place where Google would have picked it up.


robots.txt only tells spiders not to request a certain path. The search engine can still infer the content of the page from the link or surrounding text, and show it. Google usually annotates these in the SERPs with something like "Google is not displaying the contents of the page because it's been blocked with robots.txt"


An open library will be something really cool to have! Give it a though, I'm eager to see what others diagrams




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: