hyperloglockless: High-performance, concurrent cardinality estimation

51 points tomtomwombat 4 comments

I wanted to share my work on building Rust's fastest HyperLogLog.

HyperLogLogs are space efficient data structures for the "count-distinct problem", approximating the number of distinct elements in a multiset. Paper.

hyperloglockless offers a lockless concurrent HyperLogLog and a single threaded counterpart. They're simpler, faster, and more accurate than other HyperLogLog implementations:

  • 🧵 Concurrent: AtomicHyperLogLog is a drop-in replacement for RwLock<OtherHyperLogLog>: all methods take &self, so you can wrap it in Arc and update it concurrently without &mut.
  • ⚡ Fast: Designed to be fast and simple in both single and multi-threaded scenarios.
  • 🎯 Accurate: Empirically verified accuracy for trillions of elements; other implementations break down after millions.
  • ✅ Tested: Rigorously tested with loom and benchmarked.

Any feedback welcome!