Writing a UI with Floem - Impressions

I've got a DSP plugin - an Apple AudioUnit - audio effetcs for applications like Garage Band or Logic - with the DSP code written in Rust, the glue-code in Objective C and C++, and the UI in Swift using SwiftUI.

It turns out, SwiftUI is really too heavyweight for this purpose - you wouldn't think it would be, but when the machine is processing many tracks of real-time audio, it is - things like popups needing to allocate a new native window and some of the magic it implements under the hood is pretty expensive - and the UI is non-trivial - think rendering multiple mathematical curves that do not decompose trivially into Bezier curves over an adjustable window of frequency bands while overlaying the amplitudes in those frequency bands and having the whole thing be responsive, in real time.

I've been using Lapce as my daily-driver for Rust coding for months, which is the origin of Floem, and am impressed with the stability and performance of it - so I figured before I rewrite the DSP UI in Apple's older Cocoa and hope for the best, I'd give Lapce a shot, and am amount half-way through that - plus, having spent a decade plus as one of the authors of a Java IDE, including a lot of UI component work, I figured a UI framework built to solve the set of problems an IDE faces is likely to be fairly quick to understand. I'm using a git dependency on the main branch.

Floem has spots definitely a work-in-progress - once you get to non-trivial graphics stuff, you bump into stuff that just hasn't been implemented - like, try to stroke a Bezier path and you bump into PathSeg::Line(_) => todo!() in the OpenGL renderer (really? - you can stroke a quadratic curve but not a straight line??) - but these aren't hard to work around (either make everything a quad curve with the control point on the destination, or take the original Bezier path and stroke it yourself and fill that yourself - filling works fine).

I don't know that you're likely to bump into those rough edges if you're just using built-in components as opposed to writing them from scratch (I've been writing components from scratch since the 90s in Delphi, and a lot of the things this tool needs - like popup sliders that can be configured with different ranges, units and non-linear scalings - like, the ticks you print and the mapping of pixel-to-value is sometimes logarithmic or is scaled by a sine curve to make the range of values presented to the user easily reachable without micro-adjustments).

But, holy crap, the performance is stunning - and blows the SwiftUI version away \*even when running debug builds that do a bunch of logging while painting. And in terms of capability, this is sacrificing nothing, and even improving on SwiftUI in a lot of ways (for example, SwiftUI's handling of keyboard focus - what there is of it - is atrocious - clearly desktop UIs were an afterthought).

The coding paradigm is a lot like SwiftUI's - I'm not sure I'm in love with that approach in either place, but it beats imperatively drawing Java Swing components, and works. But it was fast to pick up, and with a bit of digging in the source code and examples I've been able to figure out almost everything I need (there must be a way to measure the pixel bounds of a string of text during layout without actually rendering it - in Swing, I would use an offscreen 1-pixel image's Graphics object configured to match the scaling of the target display).

All in all, despite the rough edges, it's pretty awesome, and if it works as well as it does for this, it's likely to be just fine for applications with more traditional user interfaces.

Whether I can wire the whole mess up inside a Cocoa NSView without launching an application (AudioUnits get loaded in-process in foreign applications) and have the whole thing clean itself up without leaking a byte may yet take some hackery or be a showstopper, but so far, pretty darned good.

Things that I've bumped into that need work are:

• The bezier path stroking issue (lines and cubic curves - lines trivially decompose into quad curves which are supported, and cubics can be decomposed into them with some error margin - every font rasterizer on the planet has to do this)
• Handling some of the more exotic event types (on Apple, touch events of various sorts) - lowest common denominator is inadequate
• Scroll events - a trackpad will give a ton of scroll events with deltas between -1 and 1, while a mouse wheel will give you deltas like 24, and while the underlying mouse event on Mac OS has an isPrecise parameter you can use to differentiate them, there's no differentiating them in floem - best you can do is assume let is_mouse = deltaX.abs() > 1 || deltaY.abs() > 1) and multiply by some factor.
• Accessibility support is going to be important (and is hard - anyone doing this for a UI toolkit should look at UNLV prof Andreas Stefik's Quorum language and copy what they did - his field of study is user interfaces for the blind, and making them actually work - the APIs in many OS's for this are dreadful, and "check the checkbox" accessibility support in many UI toolkits isn't actually usable by humans) - it would be a good way to leapfrog other toolkits and be usable for UIs by entities that are required by law to produce accessible applications. Not fun, but very worthwhile.
• While it's great that the text support is sufficient for a full-fledged IDE text editor, sometimes you just need to paint a caption somewhere, and there really should be a simple way to do that on PaintCx, maybe with a helper struct to configure font properties on - the API for doing this assumes you're in the business of wrapping text, maintaining scroll positions, highlighting selections, having different styling for spans of characters, sometimes you just want paint this string at x,y without it being a heroic undertaking. Also, it's incredibly non-obvious that you need to call TextLayout.shape_until_scroll() (tough to say what that method name means) or it will appear to succeed but paint nothing.

Anyway, if the question is Are we GUI yet, the answer just might be yes.