hyperloglockless: High-performance, concurrent cardinality estimation
I wanted to share my work on building Rust's fastest HyperLogLog.
HyperLogLogs are space efficient data structures for the "count-distinct problem", approximating the number of distinct elements in a multiset. Paper.
hyperloglockless offers a lockless concurrent HyperLogLog and a single threaded counterpart. They're simpler, faster, and more accurate than other HyperLogLog implementations:
- 🧵 Concurrent:
AtomicHyperLogLogis a drop-in replacement forRwLock<OtherHyperLogLog>: all methods take&self, so you can wrap it inArcand update it concurrently without&mut. - ⚡ Fast: Designed to be fast and simple in both single and multi-threaded scenarios.
- 🎯 Accurate: Empirically verified accuracy for trillions of elements; other implementations break down after millions.
- ✅ Tested: Rigorously tested with loom and benchmarked.
Any feedback welcome!