logo

Redust: a new Redis client

187 points appellation_
View on Reddit ↗

TL;DR at bottom

The Redis client library space is already pretty saturated, but I found myself frustrated with the APIs designed by the existing libraries — particularly related to pubsub.

  • redis: by far the most popular and definitely the most ergonomic library. However, I was frustrated by its pubsub support; while generally acceptable, it's impossible to perform subscription operations while consuming messages. I like the library, but the workarounds to this issue were unacceptable to me.
  • redis-async: very close to exactly what I was looking for. The high-level pubsub API is quite ergonomic and provides a low-level Tokio codec for closer interaction with the underlying TCP connection if desired. Unfortunately, the high level API is designed around 2 distinct connection types (regular and pubsub); this makes it challenging to use with connection pools.
  • simple_redis: the pubsub API is not ergonomic.
  • redis-subscribe: designed to fix the issue with the redis crate; however, it requires an entirely separate connection similar to redis-async.

I probably could have gotten redis-async working with my use case. However, I took this opportunity to design what I envisioned a simple, ergonomic library could be.

Therefore, Redust

My goal was to keep both the developer API and the internals as simple as possible. This manifested itself as a single Connection struct that implements Stream + Sink with some convenience methods for sending and receiving commands to and from Redis.

To interact with Redis, there are several options. For the most common use-case, I added a convenience method to handle sending a single command and receiving a single response. I also added a method for pipelining. Since the connection implements Stream + Sink, the developer can ignore these convenience methods and interact directly with the underlying framed TCP stream.

Pubsub is enabled by sending the subscribe command and then receiving from the connection's stream. Since the connection is not consumed by this operation, it's still available for dispatching other commands.

Obviously, part of a clean API is being able to interact with native Rust structs. This means dealing with RESP in a user-friendly way so that the developer-facing APIs don't have to worry about it.

The Elephant: RESP

RESP (REdis Serialization Protocol) is the underlying binary(ish) protocol that client libraries use to communicate with Redis.

Most Redis client libraries define their own traits for converting things to and from RESP. To be maximally ergonomic, the library developer needs to implement this trait on a wide swath of the standard library. When it comes to types exported by other crates, users of the library are SOL unless the developer has also implemented the trait on those types.

Serde seems like the obvious choice here for the best developer experience, so I spent a surprising amount of time figuring out how to implement serde for RESP (shout-out to serde-json for being a super thorough example). Most of the Rust ecosystem supports serde at some level, and serde itself handles most of the types in the standard library.

RESP does not support a some of the types available in serde, so my implementation is required to be somewhat opinionated (particularly when it comes to key-value pairing).

Finally, Something Tangible

I've published my work on Github: https://github.com/appellation/redust. If anyone is also interested in solving these kinds of problems, I'd love to receive some feedback on the code or its API.

redust:

redust-resp:

Achieving Perfection

Some things in life are perfect, but sadly this isn't one of them (yet).

  1. I'm still struggling with the best way to parse RESP from network buffers. I've been using nom (love it), but this is one of those scenarios where zero-copy isn't actually that beneficial. Currently the RESP data is parsed using zero-copy, but then immediately copied so it can be returned back to the user as a stream item. This also makes the API slightly less ergonomic, since all RESP structures have a lifetime.
  2. Deserializing RESP requires returning the remaining bytes since nom doesn't do any mutation. I'd prefer to remove the bytes from the buffer at deserialization (perhaps using Read?) rather than deal with this workaround. Combine seems like a way to potentially resolve this, but I couldn't figure out how to use it.
  3. The pubsub ergonomics are still a bit rough. Ideally there is a RAII struct to automatically unsubscribe and return the connection to a normal state when dropped, but it's not possible to do this reasonably without reference counting. I did actually try it, but the implementation was pretty pointless.
  4. The convenience functions take variations of impl AsRef<[u8]> which is nice but can lead to additional verbosity when dealing with combinations of string and binary data. Anything else is probably an antithesis to simplicity, so this might just be the way it is.

TL;DR

New Redis library. Stream + Sink-based API. Supports serde. https://docs.rs/redust