Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Wouldn't it be better, if there's an easy way, to just feed such bots shit data instead of blocking them. I know it's easier to block and saves compute and bandwidth, but perhaps feeding them shit data at scale would be a much better longer term solution.


    if ($http_user_agent ~* "BadBot") {
        limit_rate 1k;
        default_type application/octet-stream;
        proxy_buffering off;
        alias /dev/zero;
        return 200;
    }


I recommend you use gzip_static and serve a zip-bomb instead. Frees up the connection sooner and probably causes bad crawlers to exhaust their resources.


https://zadzmo.org/code/nepenthes/


No serving shit data costs bandwidth and possibly compute time.

Blocking IPS is much cheaper for the blocker.


Zip bomb?


Doesn’t that tie up a socket on the server similarly to how a keepalive would on the bot user end?


I don't think so. The payload size of the bytes on the wire is small. This premise is all dependent on the .zip being crawled synchronously by the same thread/job making the request.


What if bots catch on to zip bombs, and just download them really slowly?

https://en.wikipedia.org/wiki/Zeno%27s_paradoxes#Dichotomy_p...


Their objective is not to DDOS websites, if they catch on, they will download it fast and then discard it.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: