I just rewrote llama.cpp server in Rust (most of it at least), and made it scalable
Long story short, I rewrote most of the llama-server, made it scalable, and bundled that into Paddler.
Initially, the project started as a monitor for the llama.cpp servers, but ended up with being an entire platform for self-hosting LLMs written in Rust, with its own backend (still uses llama.cpp for inference, but the entire infra and server components are custom now).
I just released it after a few months of 10-12 hours a day coding, I am proud of it, please check it out, let me know what you think. :)