On my way to make the production rusty

TL;DR: I have ported a mqtt-to-kafka proxy Go service to Rust and thread safety is definitely easier with Rust.

Hi everyone,

This post is only to share my experience with rust, especially for people who are afraid of taking the next step.

First of all, I am a Go developer and I have been using this language personally and professionally for more than 6 years. I love this language and I will continue using it 99% of the time if I have to start a new project.

Some context at first. At my work, we process data from air quality monitors. Basically, a device connects to a mqtt broker, a service fetches data from the broker to transfer them to kafka before being sanitized and stored into a database. As the load is increasing and this data flow is showing limits, my project was to build a simple mqtt-to-kafka proxy.

The service was initially written in Go. Development was smooth, benchmarks were amazing (70k msq/sec) on the testing environment so I felt confident to deploy it in prod (with the approval of my leader of course).
1% of the devices were redirected to this new service while the rest still used the legacy software stack, no problem for 1h. Moving to 50%, still no problem as the CPU usage is still low and ram usage constant. By the end of the day, we were quite confident so we moved 100% of the traffic to this new service and the monitors were still all green but we kept a hot standby in case something went wrong. 4hours later, pager duty is telling me there is an issue and we are not receiving data at all so I have to revert my changes and use the legacy software stack with disappointment. After checking the logs, we found out the service started to behave like chrome regarding ram usage while debug didn't show anything special, so it was hard to see what went wrong.

This mqtt-to-kafka proxy must be highly asynchronous so translating this with Go was quite a challenge since it had to synchronize many goroutines per client and even more channels. More complexity was added with mutexes to keep each client's state correct whatever action is performed.

Here (finally) comes rust, I have done some projects with this language but I wanted to challenge myself to see if I can be productive with it. The mqtt broker (near) implementation was quite easy with libs but more thoughts had to be put on how to use thread safe objects within a thread safe hashmap that I can share with many tokio tasks.
I will not lie, it took me a day to come up with the right design while cargo was screaming at me every time I tried to run my code, but after all, I managed to have something 90% reimplemented in Rust from my Go codebase.

While Go avoids many errors at compile time compared with other languages like JS or Python, rust compiler does a deeper work at this level. No more unclosed channels, unclosed sockets or a mutex not unlocked. I am aware shit can still hit the fan if I am not careful with nested thread safe objects but I can "feel" my code is safe after it compiles successfully.

Even if I am really happy about my work, my biggest complain is the number of available crates that is still not as developed as Go from my point of view. Code is a bit too verbose, especially type definition and there is no consensus about async runtimes (even if tokio seems to be used most of the time). Also it took me quite a while to navigate through doc.rs without passing out. I guess the points I just raised are mostly because I am not used to the rust ecosystem but I am sure it will be better in the future. One thing is sure, Rust will definitely be my new tool for critical service development (and soon for embedded systems as my hobby)

Overall, it was a great experience and I had to share it. The learning curve is no joke but it was definitely worth the time as you can apply rust idioms to other programming languages. The best lesson I have received is "stop fighting the compiler" then life become easier.

This post is already too long so I would like to finish by thanking all this sub reddit for being welcoming, patient and sharing amazing tips.