logo

Fluvio is trending on GitHub! Thank You!

135 points drc1728
View on Reddit ↗

We have been building Fluvio open source for over 5 years. The core repo is \~120,000 lines of code.

What is Fluvio?
Fluvio is a distributed streaming system built from the ground up in Rust.

Fluvio uses topics to collect and store events in the form of an immutable distributed event log. Fluvio is used to implement enterprise service buses, message queues, and data streaming. It's like Kafka in Rust.

What is SDF? SDF, short for Stateful Dataflows, is stream processing, where you can join/split data flow and make call-outs to other services such as AI/ML for enrichment or count elements. It's like Flink + Microservices in Rust.

How is Fluvio + SDF unique? We're piloting an innovative real-time infrastructure that simplifies and enhances your data operations:

  • Seamless Data Integration: Effortlessly connect data from any source—databases, webhooks, and more—into your favorite tools.
  • Flexible Data Transformation: Customize and materialize data views programmatically or with SQL, tailored to your needs.
  • AI-Powered Enrichment: Enrich data with AI, including generative AI and machine learning, all within one platform.

And we have more bells and whistles added which would easily be over \~250k lines across the connector development kit, smart module development kit, stateful dataflow, and more. Stateful DataFlow is layer for stateful stream processing and building end to end data pipelines.

It's a robust production ready system. We'd love for the Rust teams to consider using Fluvio and Stateful DataFlow for your next project.

I wont bore you with a list of features cause there are are lot. Here are the links to the repo and the docs.