logo

[Media] lz4rip: an LZ4 codec in Rust, optimized for transfer speed

65 points event666 9 comments
View on Reddit ↗ External link ↗
[Media] lz4rip: an LZ4 codec in Rust, optimized for transfer speed

lz4rip started as a dictionary support PR for lz4_flex, then escalated into its own project. Optimized for small messages (under 8 KB) in tight loops, where latency dominates.

Performance. lz4rip trades ratio for speed via half the L1d footprint (smaller hash tables) and aggressive skip acceleration on non-matches. The chart above stacks compress + transfer @1 GB/s + decompress time across the Silesia corpus. Lower is better.

Unsafe boundary. 20 unsafe blocks across 4 files (down from 60+). All algorithm code is #[forbid(unsafe_code)], including the compressor hot loop. This encapsulation prevents issues like RUSTSEC-2026-0041. The paranoid feature compiles with zero unsafe.

vs lz4_flex. First-class dictionary support+training, no_std + no-alloc, const-generic table sizes (down to 512 B), dual-table DictCompressor, epoch-based table reuse for hot loops, and 4-crate workspace split for LLVM isolation.

no_std. Works without std or alloc. Hash tables live on the stack.

WASM. Available on JSR with dictionary support.

Tested to hell and back, fuzzed and Miri-clean on x86_64 and aarch64.

Primary consumer is OMQ.rs, a message transport that uses lz4rip for its lz4+tcp:// transport (benchmarks).

All charts (per-file, size sweep, dictionary, aarch64) are in the repo.

Heavily AI assisted, but not slop (hopefully).