logo

Rust is showing a lot of promise in the DataFrame / tabular data space

135 points MrPowersAAHHH
View on Reddit ↗

Rust has amazing data/DataFrame libraries and they're getting even better.

  • Polars is a blazing fast DataFrame library with a beautiful user interface and an awesome getting started guide. The impressive h2o benchmark results have gotten Polars a lot of users.
  • arrow-datafusion is another great DataFrame library, especially if you like running SQL queries. It's so easy to query a Parquet / CSV dataset with SQL using DataFusion. I've run local benchmarks and it's super fast. The DataFusion docs are a bit lacking, which is a shame, for such a developed and amazing library. I hope to make these better and help spread the world about how truly amazing this lib is.
  • arrow2 and parquet2 are great foundational libraries for and DataFrame libs in Rust.
  • I'm working on delta-rs which brings the power of Delta Lake to the Rust community. CSV / Parquet lakes are limited and Delta Lakes offer a ton of advantages (versioned data, time travel, ACID transactions, schema enforcement, etc). We're working to bring full Polars and DataFusion support to delta-rs, see the roadmap.

Rust is quickly becoming one of the best languages to do DataFrame workloads. It's just shockingly fast. The Rust data community still feels small, but I think it'll grow rapidly in the coming years. The libraries that are being built out are just so amazing.