Distributed computing in Rust

Hello there Rustaceans,

I am a machine learning engineer who recently took up Rust and I love it!

I think that Rust would be a great language to write data processing tools in and could replace existing libraries (usually written in Python or in C with a Python wrapper). Data processing at scale usually means distributed parallel computing. Libraries like dask and ray are amazing libraries where you can dynamically dispatch functions on a running cluster. Dask is written entirely in Python and solves this by serializing Python functions and sending them using TCP to workers' processes that runs them in a local ThreadPool. Ray is written in C++ and does the same (although it is not clear to me how they could serialize C++ functions at runtime ?).

My question here is how can I achieve the same behavior at runtime using Rust. Crates like serde_closure is used to serialize closure and send them across the network. The issue here is that the client process and worker process binaries should be the same ! You can't spawn a cluster and send arbitrary rust functions to the running processes. Do you have any insight into how this could be done ? Maybe some kind of runtime linking :?Thanks a lot for you help,