Succinctly: A fast jq/yq alternative built on succinct data structures
I've been working on Succinctly, a Rust library and CLI tool that provides jq and yq functionality using succinct data structures (semi-indexing with rank/select).
What it does:
- Covers most jq and yq query patterns (reduce, limit, recurse, regex, path functions, etc.)
- Parses JSON at ~880 MiB/s, YAML at ~250-400 MiB/s
- Supports position-based navigation (at_offset, at_position) for IDE integration
What it doesn't do (yet):
- input/inputs (streaming multiple JSON values from stdin)
- Streaming for files larger than memory
- Some advanced YAML edge cases
Performance vs jq (AMD Ryzen 9 7950X):
| Size | succinctly | jq | Speedup | succ Mem | jq Mem | Mem Ratio |
|---|---|---|---|---|---|---|
| 1MB | 25ms | 45ms | 1.8x | 5 MB | 17 MB | 0.30x |
| 10MB | 221ms | 382ms | 1.7x | 14 MB | 149 MB | 0.09x |
| 100MB | 2.3s | 3.9s | 1.7x | 104 MB | 1 GB | 0.07x |
Performance vs yq (Apple M1 Max):
| Size | succinctly | yq | Speedup | succ Mem | yq Mem | Mem Ratio |
|---|---|---|---|---|---|---|
| 1MB | 16ms | 117ms | 7.2x | 12 MB | 78 MB | 0.16x |
| 10MB | 112ms | 1.07s | 9.5x | 66 MB | 687 MB | 0.10x |
| 100MB | 1.06s | 10.3s | 9.7x | 573 MB | 6 GB | 0.09x |
Hardware optimizations:
- x86_64: AVX2 SIMD, POPCNT, BMI2 (PDEP/PEXT for DSV parsing)
- ARM: NEON intrinsics
- Benchmarks run on AMD Zen 4 and Apple M1 Max — results will vary on older CPUs without these instructions
Example usage:
succinctly jq '.users[].name' data.json
succinctly yq '.spec.containers[]' k8s.yaml
succinctly yq -o json '.' config.yaml # YAML to JSON
Why succinct data structures?
Instead of building a full DOM, semi-indexing creates a lightweight index over the raw text. This enables O(1) navigation to any node without parsing the entire document upfront — and uses 6-10x less memory than jq/yq on large files.
The library is no_std compatible.
Links:
- GitHub: https://github.com/rust-works/succinctly
- Crates.io: https://crates.io/crates/succinctly
Feedback welcome — especially bug reports for queries that work in jq/yq but fail here.