We love to see performance numbers. It is a core objective for us. We are excited at another milestone in our ongoing effort: a 4x reduction in write latency for our data pipeline, bringing it down from 120ms to 30ms! This improvement is the result of transitioning from a C library accessed through a Python application to a fully Rust-based implementation. This is a light intro on our architectural changes, the real-world results, and the impact on system performance and user experience.

Chart A and Chart B shown in the image above.

So Why did we Switch to Rust from Python? Our Data Pipeline is Used by All Services!

Our data pipeline is the backbone of our real-time communication platform. Our team is responsible for copying event data from all our APIs to all our internal systems and services. Data processing, event storage and indexing, connectivity status and lots more. Our primary goal is to ensure up-to-the-moment accuracy and reliability for real-time communication.

Before our migration, the old pipeline utilized a C library accessed through a Python service, which buffered and bundled data. This was really the critical aspect that was causing our latency. We desired optimization, and knew it was achievable. We explored a transition to Rust, as we’ve seen performance, memory safety, and concurrency capabilities benefit us before. It’s time to do it again!

We Value Highly Rust Advantages with Performance and Asynchronous IO

Rust is great in performance-intensive environments, especially when combined with asynchronous IO libraries like Tokio. Tokio supports a multithreaded, non-blocking runtime for writing asynchronous applications with the Rust programming language. The move to Rust allowed us to leverage these capabilities fully, enabling high throughput and low latency. All with compile-time memory and concurrency safety.

Memory and Concurrency Safety

Rust’s ownership model provides compile-time guarantees for memory and concurrency safety, which preempts the most common issues such as data races, memory leaks, and invalid memory access. This is advantageous for us. Going forward we can confidently manage the lifecycle of the codebase. Allowing a ruthless refactoring if needed later. And there’s always a “needed later” situation.

Technical Implementation of Architectural Changes and Service-to-Service and Messaging with MPSC and Tokio

The previous architecture relied on a service-to-service message-passing system that introduced considerable overhead and latency. A Python service utilized a C library for buffering and bundling data. And when messages were exchanged among multiple services, delays occurred, escalating the system's complexity. The buffering mechanism within the C library acted as a substantial bottleneck, resulting in an end-to-end latency of roughly 120 milliseconds. We thought this was optimal because our per-event latence average was at 40 microseconds. While this looks good from the old Python service perspective, downstream systems took a hit during unbundle time. This causes overall latency to be higher.

In Chart B above shows when we deployed that the average per-event latency increased to 100 microseconds from the original 40. This seems non-optimal. Chart B should show reduced latency, not an increase! Though when we step back to look at the reason, we can see how this happens. The good news is now that downstream services can consume events more quickly, one-by-one without needing to unbundle. The overall end-to-end latency had a chance to dramatically improve from 120ms to 30ms. The new Rust application can fire off events instantly and concurrently. This approach was not possible with Python as it would have also been a rewrite to use a different concurrency model. We could have probably rewritten in Python. And if it’s going to be a rewrite, might as well make the best rewrite we can with Rust!

Resource Reduction CPU and MemoryOur Python service would consume upwards of 60% of a core. The new Rust service consumes less than 5% across multiple cores. And the memory reduction was dramatic as well, with Rust operating at about 200MB vs Python’s GBs of RAM.

New Rust-based Architecture SampleThe new architecture leverages Rust’s powerful concurrency mechanisms and asynchronous IO capabilities. Service-to-service message passing was replaced by utilizing multiple instances of Multi-Producer, Single-Consumer (MPSC) channels. Tokio is built for efficient asynchronous operations, which reduces blocking and increases throughput. Our data process was streamlined by eliminating the need for intermediary buffering stages, and opting instead for concurrency and parallelism. This improved performance and efficiency.

Example Rust App

The code isn’t a direct copy, it’s just a stand-in sample that mimics what our production code would be doing. Also, the code only shows one MPSC where our production system uses many channels.

Cargo.toml: We need to include dependencies for Tokio and any other crate we might be using (like async-channel for events).
Event definition: The Event type is used in the code but not defined as we have many types not shown in the this example.
Event stream: event\_stream is referenced but not created in the same way we do with many streams. Depends on your approach so the example keeps things simple.

The following is a Rust example with code and Cargo.toml file. Event definitions and event stream initialization too.

Cargo.toml

[package]
name = "tokio_mpsc_example"
version = "0.1.0"
edition = "2021"

[dependencies]
tokio = { version = "1", features = ["full"] }

main.rs

use tokio::sync::mpsc;
use tokio::task::spawn;
use tokio::time::{sleep, Duration};

// Define the Event type
#[derive(Debug)]
struct Event {
    id: u32,
    data: String,
}

// Function to handle each event
async fn handle_event(event: Event) {
    println!("Processing event: {:?}", event);
    // Simulate processing time
    sleep(Duration::from_millis(200)).await;
}

// Function to process data received by the receiver
async fn process_data(mut rx: mpsc::Receiver<Event>) {
    while let Some(event) = rx.recv().await {
        handle_event(event).await;
    }
}

#[tokio::main]
async fn main() {
    // Create the channel with a buffer size of 100
    let (tx, rx) = mpsc::channel(100);

    // Spawn a task to process the received data
    spawn(process_data(rx));

    // Simulate an event stream with dummy data for demonstration
    let event_stream = vec![
        Event { id: 1, data: "Event 1".to_string() },
        Event { id: 2, data: "Event 2".to_string() },
        Event { id: 3, data: "Event 3".to_string() },
    ];

    // Send events through the channel
    for event in event_stream {
        if tx.send(event).await.is_err() {
            eprintln!("Receiver dropped");
        }
    }
}

Rust Sample Files

Cargo.toml:
- Specifies the package name, version, and edition.
- Includes the necessary tokio dependency with the “full” feature set.
main.rs:
- Defines an Event struct.
- Implements the handle\_event function to process each event.
- Implements the process\_data function to receive and process events from the channel.
- Creates an event\_stream with dummy data for demonstration purposes.
- Uses the Tokio runtime to spawn a task for processing events and sends events through the channel in the main function.

Benchmark

Tools used for Testing

To validate our performance improvements, extensive benchmarks were conducted in development and staging environments. Tools, such as hyperfine https://github.com/sharkdp/hyperfine and criterion.rs https://crates.io/crates/criterion were used to gather latency and throughput metrics. Various scenarios were simulated to emulate production-like loads, including peak traffic periods and edge cases.

Production Validation

In order to assess the real-world performance of the production environment, continuous monitoring was implemented using Grafana and Prometheus. This setup allowed for the tracking of key metrics such as write latency, throughput, and resource utilization. Additionally, alerts and dashboards were configured to promptly identify any deviations or bottlenecks in the system's performance, ensuring that potential issues could be addressed promptly. We of course deploy carefully to a low percentage of traffic over several weeks. The charts you see are the full-deploy after our validation phase.

Benchmarks Are not Enough

Load testing proved improvements. Though yes, testing doesn’t prove success as much as it provides evidence. Write latency was consistently reduced from 120 milliseconds to 30 milliseconds. Response times were enhanced, and end-to-end data availability was accelerated. These advancements significantly improved overall performance and efficiency.

Before and After

Before the legacy system, service-to-service messaging was done with C library buffering. This involved multiple services in the message-passing loop, and the C library added latency through event buffering. The Python service added an extra layer of latency due to Python's Global Interpreter Lock (GIL) and its inherent operational overhead. These factors resulted in high end-to-end latency, complicated error handling and debugging processes, and limited scalability due to the bottlenecks introduced by event buffering and the Python GIL.

After implementing Rust, message-passing via direct channels eliminated intermediary services, while Tokio enabled non-blocking asynchronous IO, significantly boosting throughput. Rust's strict compile-time guarantees reduced runtime errors, and we get robust performance. Improvements observed included a reduction in end-to-end latency from 120ms to 30ms, enhanced scalability through efficient resource management, and improved error handling and debugging facilitated by Rust's strict typing and error handling model. It’s hard to argue using anything other than Rust.

Deployment and Operations

Minimal Operational Changes

The deployment underwent minimal modifications to accommodate the migration from Python to Rust. Same deployment and CI/CD. Configuration management continued to leverage existing tools such as Ansible and Terraform, facilitating seamless integration. This allowed us to see a smooth transition without disrupting the existing deployment process. This is a common approach. You want to change as little as possible during a migration. That way, if a problem occurs, we can isolate the footprint and find the problem sooner.

Monitoring and Maintenance

Our application is seamlessly integrated with the existing monitoring stack, comprising Prometheus and Grafana, enabling real-time metrics monitoring. Rust's memory safety features and reduced runtime errors have significantly decreased the maintenance overhead, resulting in a more stable and efficient application. It’s great to watch our build system work, and even better to catch the errors during development on our laptops allowing us to catch errors before we push commits that would cause builds to fail.

Practical Impact on User Experience

Improved Data AvailabilityQuicker write operations allow for near-instantaneous data readiness for reads and indexing, leading to user experience enhancements. These enhancements encompass reduced latency in data retrieval, enabling more efficient and responsive applications. Real-time analytics and insights are better too. This provides businesses with up-to-date information for informed decision-making. Furthermore, faster propagation of updates across all user interfaces ensures that users always have access to the most current data, enhancing collaboration and productivity within teams who use the APIs we offer. The latency is noticeable from an external perspective. Combining APIs can ensure now that data is available and sooner.

Increased System Scalability and Reliability

Rust-focused businesses will get a serious boost advantage. They'll be able to analyze larger amounts of data without their systems slowing down. This means you can keep up with the user load. And let's not forget the added bonus of a more resilient system with less downtime. We're running a business with a billion connected devices, where disruptions are a no-no and continuous operation is a must.

Future Plans and Innovations

Rust has proven to be successful in improving performance and scalability, and we are committed to expanding its utilization throughout our platform. We plan to extend Rust implementations to other performance-critical components, ensuring that the platform as a whole benefits from its advantages. As part of our ongoing commitment to innovation, we will continue to focus on performance tuning and architectural refinements in Rust, ensuring that it remains the optimal choice for mission-critical applications. Additionally, we will explore new asynchronous patterns and concurrency models in Rust, pushing the boundaries of what is possible with high-performance computing.

Technologies like Rust enhance our competitive edge. We get to remain the leader in our space. Our critical infrastructure is Rusting in the best possible way. We are ensuring that our real-time communication services remain the best in class.

The transition to Rust has not only reduced latency significantly but also laid a strong foundation for future enhancements in performance, scalability, and reliability. We deliver the best possible experience for our users.

Rust combined with our dedication to providing the best API service possible to billions of users. Our experiences positions us well to meet and exceed the demands of real-time communication now and into the future.

120ms to 30ms: Python 🐍 to Rust 🦀🚀