That's why I said you need to control your allocator.
It's so much easier than rewriting everything that deals with pointers.
And of course everything has virtual memory. But even without it, you're not really worse off when you upgrade the CPU if masking was already enforced.
> This was already used on 32b x86: PAE decorrelated the physical address space from the virtual (giving the OS 36 bits physical address — with a theoretical upper bound of 64 — even as individual processes were still restricted to 32); and on windows the /LARGEADDRESSAWARE link switch allowed manipulating the virtual address space on a per-process granularity.
I wouldn't characterize this as very similar. Physical and virtual address space already had to be decorrelated to go above about 1GB, and PAE doesn't meaningfully extend how much memory a 32 bit processes can have. And LAA is based on software assumptions that were never particularly tied to hardware, and it can be relevant with or without PAE.
> That's why I said you need to control your allocator.
The allocator does not matter unless by "control your allocator" you mean "the OS provides an API which can restrict the virtual memory system", which has nothign to do with the allocator.
If you're using 19 bits for tagging and the OS returns a 52 bit pointer, you don't have the space for your tagging scheme, no matter what your allocator is.
> PAE doesn't meaningfully extend how much memory a 32 bit processes can have.
Of course not. What it does is show that there is no correlation between the physical and virtual address spaces, so you can have a 64 bits physical address space and only provide a 48 bits virtual address space to applications.
> And LAA is based on software assumptions that were never particularly tied to hardware, and it can be relevant with or without PAE.
Missing the point, the example of LAA shows that the OS can provide different virtual address spaces to different processes. So the OS can provide 48 bit virtual address spaces by default, and a full 64 bits virtual address space to applications which request it. The former can then keep using tagged pointers just fine.
The allocator gets address space. The OS doesn't just shove things into your process in general. It can't return a 52 bit pointer out of nowhere. My first draft mentioned OS explicitly but it's not really the OS that's making the decisions.
> What it does is show that there is no correlation between the physical and virtual address spaces
It does show that, but that decorrelation is neither necessary nor sufficient to solve this problem.
> Missing the point, the example of LAA shows that the OS can provide different virtual address spaces to different processes.
I agree with that. In the context of your overall post it wasn't clear to me that that was why you mentioned it.
> The OS doesn't just shove things into your process in general. It can't return a 52 bit pointer out of nowhere. My first draft mentioned OS explicitly but it's not really the OS that's making the decisions.
Of course it’s the OS making the decision. You ask the OS for memory and it returns whatever it wants.
The allocators acts as bridge between the application and OS and can do cool stuff but if the OS returns a 52 bits pointer there isn’t a thing the allocator can do about it that would result in a working 48 bits pointer.
> It does show that, but that decorrelation is neither necessary nor sufficient to solve this problem.
It absolutely is necessary, since the entire subject is to not break applications requiring a smaller address space for their tagging scheme to work even as the system and other applications migrate to larger ones.
The allocator can choose which part of the address space to fill. It does not have to deal with "whatever the OS wants".
> It absolutely is necessary, since the entire subject is to not break applications requiring a smaller address space for their tagging scheme to work even as the system and other applications migrate to larger ones.
Even if you have a 1:1 memory mapping, you can leave the applications that want small addresses at the start of memory. You don't need to decorrelate virtual and physical addresses to do that.
> I'm assuming an OS that isn't refusing to do things for no reason.
> Aren't you?
You are apparently assuming the OS would have no reason to refuse. I've seen DeRaadt cook, I'd rather not assume anything that's not guaranteed. And I especially wouldn't make up new modes of operations which don't exist for no justifiable reason.
That's why I said you need to control your allocator.
It's so much easier than rewriting everything that deals with pointers.
And of course everything has virtual memory. But even without it, you're not really worse off when you upgrade the CPU if masking was already enforced.
> This was already used on 32b x86: PAE decorrelated the physical address space from the virtual (giving the OS 36 bits physical address — with a theoretical upper bound of 64 — even as individual processes were still restricted to 32); and on windows the /LARGEADDRESSAWARE link switch allowed manipulating the virtual address space on a per-process granularity.
I wouldn't characterize this as very similar. Physical and virtual address space already had to be decorrelated to go above about 1GB, and PAE doesn't meaningfully extend how much memory a 32 bit processes can have. And LAA is based on software assumptions that were never particularly tied to hardware, and it can be relevant with or without PAE.