Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

If you always want to search by one thing, you can manually index by that thing. In your case, arrange your log files by the first 6 hex characters of the user ID (/var/log/xxx/xxx/date.log), and grep will typically only then have a few megabytes to scan.

If you need real indexes, or just want something industry standard and maintainable rather than 'some guys grep script', then elastic search is probably the way to go.



That was just one thing we needed to search for (but by far the most common). The guy that wrote the parallel grep did try creating some indexes of common fields to speed searches, but quickly realized that he was re-implementing the wheel (poorly)

Plus we made good use of Kibana dashboards for the service




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: