Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
erazor42
on May 23, 2018
|
parent
|
context
|
favorite
| on:
Command-line Tools can be 235x Faster than a Hadoo...
I closed at "Since the data volume was only about 1.75GB". (And probably would have until 500GB+)
imtringued
on May 23, 2018
[–]
500GB is still laptop SSD territory... unless you meant 500TB but that's still doable with a single 4U Backblaze Storage Pod with 60 x 10TB HDDs.
dajonker
on May 24, 2018
|
parent
[–]
Just reading 500TB from 60 Hdds will take at least 12 hours, assuming a very optimistic constant 200MB/s throughput per HDD. If you need things to go faster you won't be able to do that without resorting to distributed computing.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
