logo

Iggy.rs - message streaming platform

128 points spetz0
View on Reddit ↗

Hey all,

For the last 100 days, I've been working on my side project Iggy which is the message streaming platform (think of Kafka or RabbitMQ Streams). I haven't really found any message brokers or streams written in Rust (except Fluvio), so I've thought it might be a good fit :)

On the daily basis I mostly work with C#, dotnet, building some trading systems using highly performant messaging etc. and since I wanted to finally learn Rust for good, I've picked up this specific topic, as you can start with something easy (my first commits to the repo probably speak for themselves lol), and then make it a very complex solution.

Here are some of the features (readme copypasta):

  • Highly performant, persistent append-only log for the message streaming
  • Very high throughput for both writes and reads
  • Low latency and predictable resource usage thanks to the Rust compiled language (no GC)
  • Support for multiple streams, topics and partitions
  • Support for multiple transport protocols (QUIC, TCP, HTTP)
  • Fully operational RESTful API which can be optionally enabled
  • Available client SDK in Rust (more languages to come)
  • Works directly with the binary data (lack of enforced schema and serialization/deserialization)
  • Configurable server features (e.g. caching, segment size, data flush interval, transport protocols etc.)
  • Possibility of storing the consumer offsets on the server
  • Multiple ways of polling the messages:
    • By offset (using the indexes)
    • By timestamp (using the time indexes)
    • First/Last N messages
    • Next N messages for the specific consumer
  • Possibility of auto committing the offset (e.g. to achieve at-most-once delivery)
  • Consumer groups allowing message ordering and horizontal scaling across the consumers
  • Additional features such as server side message deduplication
  • Built-in benchmarking app to test the performance
  • Single binary deployment (no external dependencies)
  • Running as a single node (no cluster support yet)

At this point, Iggy server doesn't support clustering + replication (that's the upcoming topic). I'm also planning to work on simple UI etc.

During the last weekend, I've started writing some docs, for example, here's the getting started guide.

There's even the "unofficial" C# SDK being developed, which makes me quite happy, that someone already found this to be something interesting to play with.

More features are coming in the near future (well, at least once I'll document the SDK code).

Cheers