Looking for a cross-language communication framework between Rust and Python

38 points MajestikTangerine

Hello !

I'm looking for an alternative to HTTP-based inferfaces for cross-service communications. Why not grpc, protobuf and the likes ? Because it needs to be tailored for the following usage :

  • sending and receiving structured data (json) and large unstructured dataframes.

  • low-latency communication

  • ideally, I'd like to dispose of the dataframe while the content is beeing sent by the sending party

  • must handle reconnections and network interruptions.

  • ideally supports fan-in / fan-out to multiple endpoints.

  • can work in rust and python at a minimum.

So, obviously it's not that easy. I did a couple of PoCs using ZMQ, and it proved reliable enough. It also suits the dataframe-sending part by opening a new channel for every dataset. The only problem with ZMQ, sending large datasets has a tendency to slow down the rest of the communication channels (structured data), and that's not great.

What would you use as an alternative ?