Minarrow: Apache Arrow memory layout for Rust that compiles in < 2s

I've been working on a columnar data library that prioritises fast compilation and direct typed access over feature completeness.

Why another Arrow library?

Arrow-rs is excellent but compiles in 3-5 minutes and requires downcasting everywhere. I wanted something that:

• Compiles in <1.5s clean, <0.15s incremental
• Gives direct typed access without dynamic dispatch (i.e.,, as_any().downcast_ref())
• Still interoperates with Arrow via the C Data Interface
• Simple as fast - no ecosystem baggage

Design choices that might interest you:

• Dual-enum dispatch instead of trait objects: Array -> NumericArray -> IntegerArray<T>. Uses ergonomic macros to avoid the boilerplate.
• Compiler inlines everything, benchmarks show ~88ns vs arrow-rs ~147ns for 1000-element access.
• Buffer abstraction with Vec64<T> (64-byte aligned) for SIMD and SharedBuffer for zero-copy borrows with copy-on-write semantics
• MemFd support for cross-process zero-copy on Linux
• Uses portable_simd for arithmetic kernels (via the partner simd-kernels crate)
• Parquet and IPC support including memory mapped reads (via the sibling lightstream crate)

Trade-offs:

- No nested types (structs, lists, unions) - focusing on flat columnar data

- Requires nightly for portable_simd and allocator_api

- Less battle-tested than arrow-rs

If you work with high-performance data systems programming and have any feedback, or other related use cases, I'd love to hear it.

Thanks,

Pete

Disclaimer: I am not affiliated with Apache Arrow. However, this library implements the public "Arrow" memory layout which agrees on a binary representation across common buffer types. This supports cross-language zero-copy data sharing. For example, sharing data between Rust and Python without paying a significant performance penalty. For anyone who is not familiar with it, it is a key backing / foundational technology behind popular Rust data libraries such as 'Polars' and 'Apache Data Fusion'.