My program spends 96% in `__memset_sse2`.
I am working with some large-ish (2-3 GB) compressed files and I wrote the following code using the lz4 crate:
fn read<P: AsRef<Path>>(path: P) -> Result<Vec<u8>> {
let contents = std::fs::read(&path)?;
let mut decoder = lz4::Decoder::new(&contents[..])?;
let mut contents = Vec::new();
std::io::copy(&mut decoder, &mut contents)?;
Ok(contents)
}
This works OK in my computer (3 seconds per file), but when I run it in a remote server that I have access to, it takes about 50 seconds per file.
I ran cargo flamegraph to see if there is anything there, and I noticed that my program is spending 96% of its time in something called __memset_sse2 <- core::intrinsics::write_bytes <- std::io::readbuf::BorrowedCursor::ensure_init <- std::io::default_read_buf.
Running in my computer I don't see this. I see just 2% in core::instrinsics::copy_nonoverlapping <- alloc::vec::Vec::append_elements. The rest 98% (basically the 3 seconds) actually doing the decompression.
Is there a better way of writing my code to avoid this? I don't really understand any of the above, but from a bit of googling I think it means that the remote server struggles to allocate the 3 GBs for the receiving vector? There is plenty of RAM available in that server (120+ GB).
Sorry if the above is not enough information to make some suggestions, I don't have any experience with these kind of things, so I am not sure about what to look into.
Thanks.
EDIT:
Running rustup update in the remote server fixed this.
rustc from 1.73.0 to 1.75.0