Storing types in rust

I'm currently reimplementing ray tracing in one weekend with the focus on speed. One crucial performance loss is the following design, which is described by the book

trait Hittable{}
struct World {
    container: Vec<Box<dyn Hittable>>
}

Instead of adding so many indirection, I'd prefer to store each implementation of Hittable in a continuous block of memory. I would roughly prefer following layout:

struct World {
container: HashMap<TypeId, Vec<Any>>
}

which doesn't work because Any is a trait. If someone is aware if such approach is possible in rust I'd be glad for hints!

I'm fully aware that the rusty approach would most likely using a big enum storing each variant of Hittable. This would be tricky in my case since I want external implementation of Hittable to be possible; also a Hittable might vary largely in size.

​

Edit: Thank you for the helpful replies!

Learned quite a bit :)