Learnings from making things fast

Context: I've been iterating on my side project pcodec (a codec for columns of numerical data) and have gradually improved decompression speed from \150MB/s to \1GB/s. Not everything here is novel or Rust-specific, but here's what I've learned in the process:

• Branches (if statements, while loop conditions) with 50% probability are terrible for performance. I originally had such an if statement, which would sometimes save a single bit per number. But turning it off (and incorporating that fact into the compression strategy) was a clear tradeoff win, saving \~7ns/byte.
• Branches with near 0% or 100% probability are pretty harmless. The CPU's branch predictor handles these well. I still have a couple of these in the code, but removing them only seems to save \~0.5ns/byte, not worth it.
• Make as much as possible fit into cache. If you have a table that you need to repeatedly lookup from, try to ensure it fits into (say) 16kB so it will comfortably stay in L1 cache on most machines.
• SIMD instructions are only useful if you're pulling data from L1 or L2 cache. When pulling data from L3 cache or further, you'll be entirely bottlenecked on memory bandwidth instead of instructions. The implication: if you need to do a sequence of SIMD operations on your data, iterate by chunk (literally for chunk in my_slice.chunks(chunk_size) {...} if applicable) and do all your computations on that chunk at once instead of doing multiple passes.
• When your hot loop can be faster in some cases, use generics and const generics to define each important variation. Then use an upfront match clause to select the best loop. This avoids duplicating logic while achieving the performance of separate hand-written loops. It does increase binary size, but that only becomes an issue if your code is bulky to begin with or you consider too many compiled combinations.
• Learn from the literature. People have made surprisingly recent advances in topics as basic as sorting and entropy coding. I was initially skeptical of asymmetric numeral system (ANS), since everyone only talks about the compression ratio, but in my case it wouldn't improve over Huffman + run length encoding much. But given the same first-class compression ratio, it decompresses much faster and actually simplified the code.