`shimmy` Rust inference server uses bindings to a C library... and runs Python scripts in the shell
A post came up this morning: Rustacean working on local LLMs inference, it's called "shimmy".
Safe tensors running in a safe language? Too good to be true! ^((foreshadowing is a literary device in whi)
The project is open-source so I dug in.
In Cargo.toml can be spotted two inference backend features: "huggingface" and "llama"
It pulls in the llama-cpp-2 crate for its "llama" features. Oh, that crate has a disclaimer:
"This crate is not safe. There is absolutly ways to misuse the llama.cpp API provided to create UB [...]"
Not great, but it's fine as long as the implementation is sound.
For huggingface... No crates with that name. Huggingface isn't even the name of an existing inference engine, that's the name of the organization that makes transformers for Python.
Ah, /src/engine/huggingface.rs contains the actual inference engine. Let's take a look--
My jaw dropped when I discovered that the "tiny 5MB executable" produced by this source code is partially a glorified bash script for running a Python script that uses huggingface transformers.
Meanwhile the actual "MoE offload" bit is a standard llama.cpp feature ? Which is a C project ???
It got 140 upvotes on this sub. Help.
https://media1.tenor.com/m/2Io5s8jcmrUAAAAC/facepalm-hopeless.gif
