Rust to .NET compiler backend - a small update

Hello! Author of rustc_codegen_clr here, with a small-ish update.

I have made some progress working on the project, but I currently do not have time to write a whole article. Still, I thought a quick update might be nice.

Under my last article, some people asked if I had looked into mixed mode assemblies (they contain CIL(.NET IR) and native code). While I could not figure out a way to make them yet, I have implemented a feature that is a step towards that.

When NATIVE_PASSTHROUGH is enabled, the custom linker written for the project will bundle native code from object files into a shared library, and emit information necessary for native interop. So, if your code uses a C library, it will be automatically bundled with the Rust code compiled for the .NET runtime.

This work also means that Rust code compiled for the .NET runtime will be able to use shared libraries in general - the linker will automatically generate information required for the runtime to load them.

Sadly, this feature is currently Unix-only (since it uses nm to get symbols from libraries and gcc to bundle them), and only supports the C ABI.

I did check if it would work with the Rust ABI, but sadly - no. It would be great if it did (since then you could mix highly optimized native code and .NET-specific code), but it does not.

It currently causes an unwind deep within the native version of std, and, since there is nothing to catch it, it reaches the runtime managed code and causes it to crash. I have a few theories as to why that happens:

1. The "native" std is not properly initialized, since the execution starts within the runtime and skips some stages that would normally happen in native code. This one is probably the "best" one, since it should be entirely fixable.
2. A layout bug in code compiled for .NET causes weird stuff to happen - I am still fighting with type layouts, and it is entirely possible I messed something up there. Not too bad, since this is something I will fully fix at some point in time.
3. The Rust calling convention is far more non-standard than I expected - entirely possible, albeit I do not think this is the cause. Currently, the convention I used on the FFI boundary is basically "SysV calling convention with Rust type layout". I do not know much about the internal Rust calling convention, so it is possible this is entirely wrong. Depending on how different the actual convention is, this may sadly mean that mixed-mode assemblies are not possible.

Besides that, I am still trying to fix up type layouts. Turns out, they are quite tricky to get right, and cause bizarre stuff to happen when they are wrong. I think I have finally mostly fixed enum layouts.

It turns out the "C mode", the ability for the project to output C code, is shockingly good for debugging. It was originally made as a silly proof of concept, yet it is still quite useful.

I can just hook gdb up to the compiled C code, and roughly see where everything goes wrong.

This obscure feature seems to be doing such a good job, I might need to develop it further, not to actually compile Rust to C, but just to speed up the debug process.

I have also finally fixed the issue which caused MaybeUninit to break. This one was a real head scratcher, but turned out to be very stupid in the end. Due to some code copied from enum handling, I was always trying to set the field 0 of a MaybeUninit(which is a union). Since the field 0 of MaybeUninit is a () type, a write to it was removed (since setting the value of a zero-sized type causes some weird stuff to happen in .NET). Because the write was removed, the MaybeUninit was never initialized, which was quite confusing.

On a final note - I am in the process of submitting this project to Rust GSoC! I am still currently struggling with finding a right mentor. Sadly, there is very little overlap between the people who work on the Rust compiler, and people who work on .NET related stuff. I have already found some really nice people willing to mentor the project regardless, but I am still looking for a better fit.

The GSoC is not a given, since the competition is fierce, with many great ideas, but I am keeping my fingers crossed.

If you have any questions/suggestions/feedback, feel free to leave it here. I usually try to respond to all of it.