Getting a pointer to a field of an enum

I'm trying to initialize fields of an enum variant one by one, and I'm having trouble doing it soundly. I have a working implementation, but it causes UB (Miri doesn't like it).

Here is the de-macro'd code:

\t\t\tlet cff_data_ptr = addr_of_mut!(*match &mut *outline_builder_ptr {
\t\t\t\tOutlineBuilder::Cff { cff_data, .. } => cff_data as *mut Cow<'a, [u8]>,
\t\t\t\t_ => unreachable_unchecked()
\t\t\t});

Notice that it has to take an &mut reference in order to match on it. This is the UB.

Is there any way to do this without creating a reference? I'm trying to get a pointer to the cff_data field in the uninitialized memory.

And before you comment about it: yes, I am setting the discriminant correctly afterwards. After initializing all the fields, I read them back into a fully constructed instance of the enum which I then write back, which fills in the discriminant.

There is a StackOverflow answer from 2017 that describes this scenario, I am unsure if the situation has changed since then.