Motivation for behaviour of uninhabited types

During the latest Crust of Rust about FFI, Jon introduces empty enums like the following:

enum SomeExternalThing {}

I understand the utility - by defining a type you cannot instantiate (since there are no valid enum variants), you can represent an opaque type from the other side of an FFI boundary. Fair enough.

However in things like the nomicon (https://doc.rust-lang.org/nomicon/exotic-sizes.html) and in discussions of the never ! type, we see that the compiler uses the fact that ! cannot be instantiated to do optimisations - if it can't have been made, we don't have to consider any case where it would exist.

BUT this doesn't hold when the uninhabited type is behind a pointer:

One final subtle detail about empty types is that raw pointers to them are actually valid to construct, but dereferencing them is Undefined Behavior because that wouldn't make sense.

I was wondering if someone can explain the reasoning behind this. If the compiler thinks that ! can't exist and needn't be considered, why does it think any different for *const !? I understand that it's convenient for the FFI example that prompted all this, but it seems kinda contradictory.