[Media] xuniq: I built a deduplicator that's 10x faster than sort | uniq

Project link: https://github.com/lalitshankarch/xuniq

Crate link: https://crates.io/crates/xuniq

I have been dabbling with Rust recently, and wanted to build a CLI tool to replace the common use case of sort | uniq.

For a mixed workload of 5 million lines of input (50% data where each line is duplicated + 50% data where each line is unique), the results were quite promising!

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
./target/release/xuniq < data.txt > /dev/null	528.9 ± 18.9	499.1	552.2	1.00
ripuniq < data.txt > /dev/null	1016.2 ± 17.2	996.7	1046.6	1.92 ± 0.08
huniq < data.txt > /dev/null	2835.4 ± 19.0	2807.5	2863.9	5.36 ± 0.19
`sort data.txt	uniq > /dev/null`	5669.4 ± 42.2	5620.8	5749.2
awk '!seen[$0]++' data.txt > /dev/null	5484.4 ± 27.1	5441.1	5533.7	10.37 ± 0.37
perl -ne 'print if !$seen{$_}++' data.txt > /dev/null	2515.7 ± 20.2	2487.7	2557.5	4.76 ± 0.17

Internally, the code uses the XxHash3_128 algorithm which has an astronomically low rate of hash collisions (A 50% collision probability requires hashing ~2^(64) lines). So, for any realistic workload, it's collision free in practice.