The One Billion row challenge in Rust (5 min -> 9 seconds)

252 points theprophet26

Hey there Rustaceans,

I tried my hand at optimizing the solution for the One Billion Row Challenge in Rust. I started with a 5 minute time for the naive implementation and brought it down to 9 seconds. I have written down all my learning in the below blog post:

https://naveenaidu.dev/tackling-the-1-billion-row-challenge-in-rust-a-journey-from-5-minutes-to-9-seconds

My main aim while implementing the solution was to create a simple, maintainable, production ready code with no unsafe usage. I'm happy to say that I think I did achieve that \^\^

Following are some of my key takeaways:

  1. Optimise builds with the --release flag
  2. Avoid println! in critical paths; use logging crates for debugging
  3. Be cautious with FromIterator::collect(); it triggers new allocations
  4. Minimise unnecessary allocations, especially with to_owned() and clone()
  5. Changing the hash function, FxHashMap performed slightly faster than the core HashMap
  6. For large files, prefer buffered reading over loading the entire file
  7. Use byte slices ([u8]) instead of Strings when UTF-8 validation isn't needed
  8. Parallelize only after optimising single-threaded performance

I have taken an iterative approach during this challenge and each solution for the challenge has been added as a single commit. I believe this will be helpful to review the solution! The commits for this is present below:
https://github.com/Naveenaidu/rust-1brc

Any feedback, criticism and suggestions are highly welcome!