Crazy bug caused OOM in our streaming engine, can you find it?
At Epsio we're building an incremental query engine in Rust, which does a lot of "data moving"-
We had a super interesting bug around a year ago that was caused by some unexpected behaviour in Rust, wrote the answer below :)
let mut all_vecs = vec![];
let mut total_count = 0;
loop {
let new_vec = receiver.recv()?; // Incoming vecs are guaranteed to be maximum 100k items
// This can potentially heavily filter the vec
let filtered_new_vec = new_vec.into_iter().filter(|x| /* A user defined filter here */).collect();
total_count += filtered_new_vec.len()
all_vecs.push(filtered_new_vec);
if total_count > 100_000 {
// Make sure we don't keep too much memory; flush!
// ... send the vec somewhere
total_count = 0;
all_vecs = vec![];
}
}
Since then, we've heavily changed how we move data around, but thought it was worth the post.
I'll write the answer as a spoiler:
!So first, just as background- a vecs total size is decided by capacity rather than len. Usually these are closely correlated. But here, Rust tries to deliver an optimization- it uses the incoming allocation , new_vec, for filtered_new_vec. So, even if a vector filters from 100K items to just 10, it still holds the memory for 100K items. This means we could be in a situation where we think we're holding 50 items in memory, where in reality we're holding the space of milions!!<