It's notable the Apple devices are very low-RAM compared to similar devices from competitors.
Part of that is that Apples software team uses more efficient languages like (eg. Objective-C vs Java). Part of that is that applications on iOS don't have to target a huge variety of screen resolutions (and therefore are frequently loading then downscaling high res textures). Part of that is that RAM doesn't get much cheaper if you buy at Apple-scale - so a RAM bump represents a bigger hit to margins than adding other features.
But all of that comes back to bite when using LLM's, which inherently gobble RAM. And any memory saving techniques used will simply allow a competitor with more RAM to squeeze in an even bigger better smarter model.
Add to that, that you can't upgrade RAM in most Desktop Macs anymore.
I want to buy a Mac soon, and I'm really struggling to decide how much RAM I should order. Unfortunately, my budget is limited. If it wasn't, I would probably go for at least 32GB. I'm still hoping Apple might change their RAM pricing, but probably in vain.
> I want to buy a Mac soon, and I'm really struggling to decide how much RAM I should order.
I'd recommend getting at least 32GB if you're on the fence. Not being able to upgrade it is a bummer, and your future self will thank you for getting the most you possibly can.
For my most recent upgrade I went for 64GB (previously 32GB) and I'm really glad I did, especially since llama.cpp became a thing shortly after getting it.
Also in the “glad I got 64gb” camp - even though it seemed ridiculous when I bought it, technology has advanced so quickly that now it’s actually very useful.
Now I wish I’d bought 4tb rather than 2tb hard drive lol but that’s just me being lazy - that upgrade definitely felt like a step too far.
64GB of memory will struggle on 65-70B (or bigger) models, and you'll be limited to running 70B on Q3 or Q4 if you want to use it somewhat comfortably.
lmkd (low memory killer daemon) works fairly differently off of a different set of signals and different policy. But yes, conceptually they try to achieve the same goal.
I also do not know if Android combines system libraries into one big file for the savings, something Apple devices do.
The only thing that keeps me on a Mac is familiarity, and air macs are silent. I am open to any suggestions for Linux laptops that are quiet or almost silent, most have fans that rev up, I'll gladly sacrifice some CPU for quiet or even a quiet mode (easy switch on/off). Nothing I've seen matches the silence. I'm more than happy to hear anything that proves me wrong. I would be glad to hear about something like that, obviously it has to have plus like either cheaper/replaceable ram. Furthermore, I mostly use my Mac air as a remote terminal to web based services and my Linux server that I use for compiling bigger projects and home/self-hosting.
Not sure if this is the right take. Apple is betting that in the long term that flash memory will be equivalent to RAM with the right CPU / GPU architectures. The timeline is pressed up, certainly, but I don't think their thesis is wrong.
Part of that is that Apples software team uses more efficient languages like (eg. Objective-C vs Java). Part of that is that applications on iOS don't have to target a huge variety of screen resolutions (and therefore are frequently loading then downscaling high res textures). Part of that is that RAM doesn't get much cheaper if you buy at Apple-scale - so a RAM bump represents a bigger hit to margins than adding other features.
But all of that comes back to bite when using LLM's, which inherently gobble RAM. And any memory saving techniques used will simply allow a competitor with more RAM to squeeze in an even bigger better smarter model.