The One Billion row challenge in Rust (5 min -> 9 seconds)
Hey there Rustaceans,
I tried my hand at optimizing the solution for the One Billion Row Challenge in Rust. I started with a 5 minute time for the naive implementation and brought it down to 9 seconds. I have written down all my learning in the below blog post:
My main aim while implementing the solution was to create a simple, maintainable, production ready code with no unsafe usage. I'm happy to say that I think I did achieve that \^\^
Following are some of my key takeaways:
- Optimise builds with the
--releaseflag - Avoid
println!in critical paths; use logging crates for debugging - Be cautious with
FromIterator::collect(); it triggers new allocations - Minimise unnecessary allocations, especially with
to_owned()andclone() - Changing the hash function,
FxHashMapperformed slightly faster than the coreHashMap - For large files, prefer buffered reading over loading the entire file
- Use byte slices (
[u8]) instead of Strings when UTF-8 validation isn't needed - Parallelize only after optimising single-threaded performance
I have taken an iterative approach during this challenge and each solution for the challenge has been added as a single commit. I believe this will be helpful to review the solution! The commits for this is present below:
https://github.com/Naveenaidu/rust-1brc
Any feedback, criticism and suggestions are highly welcome!