> without doing anything hugely inefficient, such as initializing the full buffer
Is this so inefficient? If your code is very sensitive to IO throughput, then it seems preferable to re-use buffers and pay the initialization once at startup.
Some years ago, I needed a buffer like this and one didn't exist, so I wrote one: https://crates.io/crates/fixed-buffer . I like that it's a plain struct with no type parameters.
It can be. If you have large buffers (tuned for throughput) that end up fulfilling lots of small requests for whatever reason, for example. And there's always the occasional article when someone rediscovers that replacing malloc + memset with calloc can have massive performance savings thanks to zeroing by the OS only occuring on first page fault (if it ever occurs), instead of an O(N) operation on the whole buffer up front.
Which, if in the wrong loop, can quickly balloon from O(N) to O(scary).
If I'm reading that log-log plot right, that looks like a significantly worse than 100x slowdown on 1GB data sets. Avoiding init isn't the only solution, of course, but it was a solution.
> then it seems preferable to re-use buffers
Buffer reuse may be an option, but in code with complicated buffer ownership (e.g. transfering between threads, with the thread of origination not necessarily sticking around, etc.), one of the sanest methods of re-use may be to return said buffer to the allocator, or even OS.
> and pay the initialization once at startup.
Possibly a great option for long lived processes, possibly a terrible one for something you spawn via xargs.
Is this so inefficient? If your code is very sensitive to IO throughput, then it seems preferable to re-use buffers and pay the initialization once at startup.
Some years ago, I needed a buffer like this and one didn't exist, so I wrote one: https://crates.io/crates/fixed-buffer . I like that it's a plain struct with no type parameters.