SwapVec - A Vector that swaps to disk if growing large
I had some fun the last two days and decided to share it with you.
It solves a problem I had at work.
Actually it doesn't, because we don't use Rust.
But it would have solved it, if we would have used Rust.
So, what does it do?
It's like a very limited version of a vector, which can grow really big.
First you push items, possibly a lot, and after a certain threshold
(32Mio, 256MiB for SwapVec<u64>) it starts swapping to a temporary file.
Why would I want that?
It's for corner cases, if you want to construct data first and consume it later, without holding everything in memory.
Imagine multiple threads slowly producing giant vectors of data, passing it to a single fast consumer, while keeping a low memory profile.
Or a CSV upload of multiple gigabytes to an HTTP server, in which you want to validate every line, while uploading (abort before uploading entire file), without directly starting a long lived database transaction or keeping everything in memory. (This was the issue at work.)
Technicalities
The swap threshold and batch size can be configured.
It is a thin wrapper around de/serialization of items into and from a temporary file. However, it still offers a clean abstraction and already does things like checksums to verify everything is read back correctly.
Things like compression, async support and further optimizations are planned.
It's supposed to solve this very specific problem good and reliable.
If I think it is production ready, I will bump it to 1.0.
There is a lot of low hanging fruit. And due to it's simplicity it's very beginner friendly. So I'd like to invite you to participate, even if you have very little experience in Rust.