If it _needs_ to be in RAM, then either you got a big enough machine (then by de... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		groestl on March 3, 2019 \| parent \| context \| favorite \| on: What you need may be “pipeline +Unix commands” onl... If it _needs_ to be in RAM, then either you got a big enough machine (then by definition it's not Big Data) or it's impossible. If you manage to come by with RAM using smart algorithms, although the full dataset would never fit in RAM, then it's Big Data. So I'd argue, stream processing is Big Data, exactly because it doesn't rely too much on RAM.

Aeolun on March 3, 2019 [–]

True, but I can stream process 20GB of data on my tiny 2GB RAM home server as well.

That’s not really ‘big data’ in my opinion.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact