Looking for a cross-language communication framework between Rust and Python
Hello !
I'm looking for an alternative to HTTP-based inferfaces for cross-service communications. Why not grpc, protobuf and the likes ? Because it needs to be tailored for the following usage :
sending and receiving structured data (json) and large unstructured dataframes.
low-latency communication
ideally, I'd like to dispose of the dataframe while the content is beeing sent by the sending party
must handle reconnections and network interruptions.
ideally supports fan-in / fan-out to multiple endpoints.
can work in rust and python at a minimum.
So, obviously it's not that easy. I did a couple of PoCs using ZMQ, and it proved reliable enough. It also suits the dataframe-sending part by opening a new channel for every dataset. The only problem with ZMQ, sending large datasets has a tendency to slow down the rest of the communication channels (structured data), and that's not great.
What would you use as an alternative ?