Fearless SIMD in Brotli: my experience

I'm building a Rust tool for compressing frontend assets, and Brotli has been my main pain point. Gzip and zstd are fast enough. Brotli… not so much.

So I started digging into Dropbox's rust-brotli.

It already had a SIMD feature, but it depends on nightly Rust, which wasn't really an option for me. I ended up forking the crate and replacing that path with fearless_simd.

That alone gave me around a 5% speedup, on stable Rust and without adding unsafe code.

Then I made the mistake of thinking: well, since I'm already here...

I sprinkled hotpath-rs over the encoder, found a few more expensive paths, and started optimizing those too.

The end result is roughly 10–15% faster Brotli compression in the cases I've tested — still on stable Rust, still safe Rust, and with byte-identical compressed output.

I opened a PR upstream and also published the fork as simd-brotli:

https://crates.io/crates/simd-brotli

Code / changelog with all the nerdy details:

https://github.com/Mnwa/rust-brotli/blob/master/CHANGELOG.md

And this rabbit hole went a bit deeper than SIMD.

I also figured out a way to make compression of a single Brotli file use the whole Rayon thread pool, instead of parallelizing only across multiple files.

If anyone's interested, I can write a follow-up about how that works. It was probably the more fun optimization of the two.