logo

[Media] zrip: a from-scratch Zstd codec in pure Rust, optimized for transfer speed

175 points event666 48 comments
View on Reddit ↗ External link ↗
[Media] zrip: a from-scratch Zstd codec in pure Rust, optimized for transfer speed

After lz4rip, I decided to attempt a from-scratch implementation of Zstd, targeting the same niche: high-speed compression for data transfers, not archival storage. The result is zrip.

Scope: Levels -7 through 4 (Fast and DFast strategies). Levels above 4 add a lot of complexity for compression ratios that only matter in storage. The whole thing is ~12k lines of Rust.

Dictionaries: Full COVER and FastCOVER dictionary training is built in. Useful for small-message workloads.

Performance: zrip is significantly faster than ruzstd (L1 only) and edges out structured-zstd (~600 unsafe blocks). Even zrip's paranoid feature (pure safe Rust, zero SIMD) beats ruzstd by 2x. C zstd is still king on both throughput and ratio (see scatter chart), but zrip is the fastest pure-Rust option.

All charts (per-file pipeline, scatter, matrix) are in the repo. The chart posted here stacks compress time + transfer time @100MB/s + decompress time. So lower is better.

Unsafe boundary: All algorithm and control-flow code is #![forbid(unsafe_code)]. Unsafe is confined to small primitives modules (unchecked indexing, unaligned reads, SIMD intrinsics) with debug_assert! guards. The paranoid feature compiles pure safe Rust with zero SIMD.

no_std + WASM: Works with no_std + alloc. Also available as a JSR package for WASM, where it's 15% faster encode and 14% faster decode than C zstd compiled to WASM.