ELI5: How is the memory layout of HashMap not a bad idea?

I'm somewhat new to programming, so maybe I'm missing something obvious, but when looking at the code of hashbrown, std::collections::HashMap seems to be grown in both directions in memory, the fast lookup table being extended forwards, but the table holding the actual elements backwards from the origin pointer.

How does the compiler avoid datastructures just growing into each other?