Running inference locally on the new Google's Gemma 2 2B models with Rust
Hey r/rust. Two weeks ago, as a weekend project/excuse to try the Rust language, I created the most minimal code (without ML libraries) that lets you run inference locally (CPU) on the new Google's Gemma2 chat models with Rust. Since then I added more things to the project: int8 quantization, SIMD matmul, 9B model support, and even a light WebUI with markdown and syntax highlighting to interact with the Rust backend. Now the 2B quantized model runs at 12 tok/s on my 8-core laptop, and the 9B runs at 4 tok/s (20 tok/s and 8 tok/s on a 16-core machine). If you want to contribute with some optimizations or other changes, feel free to do a pull request. A demo video is available in the main repo.
Main Repo: https://github.com/samuel-vitorino/lm.rs
WebUI repo: https://github.com/samuel-vitorino/lm.rs-webui
