Spent an afternoon on a perf issue that 56 bytes of padding fixed

Three atomic fields in a struct. Each written by a different thread. No locks. Added more cores… and performance got worse....perf stat looked fine.
Flamegraph looked fine too, the hot function was exactly where I expected it to be. I spent a while blaming the scheduler before finally checking the struct layout,turns out all three atomics were sitting on the same cache line.

The cores were basically fighting over the same cache line, so the slowdown was disappearing into cache coherency traffic inside the CPU. Normal profiling barely showed it. I added padding between the fields and throughput improved by 5x.

What surprised me most is that I already knew about false sharing in theory, I’d just never actually hit it in production code before. It’s one of those hardware-level effects that’s easy to forget until it suddenly dominates performance.

I wrote up a short explanation of what was happening here:

I also built a small tool that computes repr(C) offsets and warns when atomics share a cache line: