Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

This old article has the same topic with a more complex example and a surprising result about parsing 3.46Gb of data:

https://adamdrake.com/command-line-tools-can-be-235x-faster-...



A few years ago I had a personal project where I needed to load some of Wikipedia's tables into MySQL on a cheap computer. On my first attempt to just load the downloaded SQL scripts, the computer churned for 3 days before I called it a quits on the attempt. This article inspired the alternative approach: strip the first few lines down until the first Insert collection of inserts, use sed to convert '),(' to \n, then use sed again to strip the first opening and the last closing parens leftover from the earlier operation. Now we have plain CSV. Import it into MySQL. The whole operation took about an hour or two, including importing into MySQL. Yes, I lost the indices, but it was a prototype, so nothing much was lost.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: