Finally broke the scaling wall on my Rust physics engine (1.83x speedup on a single massive 4000-body island). Here’s how I fixed my threading bottlenecks.
Hey everyone,
I've been building a custom multi-threaded rigid-body physics engine in Rust (TitanEngine) from scratch, and recently I hit a massive brick wall. Under heavy load, my 8-core CPU was yielding completely abysmal scaling (less than 1.0x).
Parallelizing a bunch of separated, isolated islands is easy enough, but I was stress-testing a single massive dependency chain—a 4000-body dense stack with 3,520 constraints in one single graph.
After weeks of pulling my hair out, tracing logs, and hardware profiling, I finally managed to dismantle the bottlenecks and hit a deterministic 1.83x speedup. Here is what was actually killing my performance and how I fixed it:
1. The False-Sharing Nightmare I realized my solvers were directly modifying dense arrays inside a parallel loop. Even though threads were manipulating distinct indices (so no data corruption), the contiguous memory addresses forced them to sequentially lock equivalent cache-lines. The invisible bus stalls were insane. Fix: I transitioned the internal constraint resolutions to proxy through a padded struct utilizing #[repr(align(64))]. By committing memory states strictly outside the threaded boundaries, the false sharing completely vanished.
2. Ditching Rayon's overhead for a Crossbeam Barrier The biggest headache was the Temporal Gauss-Seidel (TGS) solver. TGS requires strict color batching. Rayon was being forced to violently spawn and tear down par_chunks iterators 150 times per substep. The stop-and-go thread execution overhead was actually taking longer than the SIMD math itself. Fix: I completely inverted the multithreading loop. Now, I generate a persistent crossbeam::scope of 8 OS threads once per Island. They navigate all the colors and iterations internally using a lock-free allocator and a double std::sync::Barrier. Thread spin-yields and CAS retries instantly dropped from 190+ million to literally 0.
3. Sequential blocks killing Amdahl's Law Before Rayon could even dispatch workers, my single-threaded setup functions were bottlenecking everything. Fix: I dismantled the single-threaded graph-coloring array copy and replaced it with a lock-free Multi-Threaded Prefix-Sum scan (distributing O(N) writes fully across 8 workers). I also replaced a massive CAS spin-lock on my penetration accumulator with a local map-reduce sum() algorithm.
4. The Telemetry Proof (Tower of Silence Benchmark) To make sure my math wasn't diverging, I dumped the telemetry into a pandas dataframe (I've attached the graph to this post).
- Without Warm-Starting: 10 GS iterations failed to push forces up the stack. Deep penetrations triggered massive Baumgarte stabilization bias, exploding the kinetic energy to 1335.87 and blowing the stack apart.
- With Double-Buffered SIMD Cache: The memory hit rate jumped straight to 80%. TGS impulses warm-started perfectly. Kinetic energy capped at 44.39, decayed exponentially, and by frame 595 hit absolute rest (1.35e-09 kinetic energy with exactly 0.0 solver error).
I also got a thermodynamic sleeping protocol working that cleanly extracts dead constraint islands from the active queue when entropy hits exactly 0.0.
The Final Result:
Scene Setup: 4000 Bodies
Max Constraints in a Single Island = 3520
1 Worker Duration: 11.83s
8 Worker Duration: 6.45s
Speedup Factor: 1.83x
Getting near 2.0x on a single dense island feels like a huge milestone for this project. Next up, I need to implement a dynamic dispatch threshold (falling back to a single thread for micro-workloads under 1000 constraints, as the barrier overhead completely dominates the math at that scale).