TL;DR: 42B parameter models now run on 8GB GPUs

I just released Shimmy v1.7.0 with MoE CPU offloading, and holy shit the memory savings are real.

Before: "I need a $10,000 A100 to run Phi-3.5-MoE"
After: "It's running on my RTX 4070" 🤯

Real numbers (not marketing BS)

I actually measured these with proper tooling:

• Phi-3.5-MoE 42B: 4GB VRAM instead of 80GB+
• GPT-OSS 20B: 71.5% VRAM reduction (15GB → 4.3GB)
• DeepSeek-MoE 16B: Down to 800MB with aggressive quantization

Yeah, it's 2-7x slower. But it actually runs instead of OOMing.

How it works

MoE (Mixture of Experts) models have tons of "expert" layers, but only use a few at a time. So we:

1. Keep active computation on GPU (fast)
2. Store unused experts on CPU/RAM (cheap)
3. Swap as needed (magic happens)

Ready to try it?

# Install (it's on crates.io!)
cargo install shimmy

# I made a bunch of optimized models for this
huggingface-cli download MikeKuykendall/phi-3.5-moe-q4-k-m-cpu-offload-gguf

# Run it
./shimmy serve --cpu-moe --model-path phi-3.5-moe-q4-k-m.gguf

OpenAI-compatible API, so your existing code Just Works™.

Model recommendations

I uploaded 9 different variants so you can pick based on your hardware:

• Got 8GB VRAM? → Phi-3.5-MoE Q8.0 (maximum quality)
• 4GB VRAM? → DeepSeek-MoE Q4 K-M (solid performance)
• Potato GPU? → DeepSeek-MoE Q2 K (800MB VRAM, still decent)
• First time? → Phi-3.5-MoE Q4 K-M (best balance)

All models: https://huggingface.co/MikeKuykendall

Cross-platform binaries

• Windows (CUDA support)
• macOS (Metal + MLX)
• Linux x86_64 + ARM64

Still a tiny 5MB binary with zero Python bloat.

Why this is actually important

This isn't just a cool demo. It's about democratizing AI access.

• Students: Run SOTA models on laptops
• Researchers: Prototype without cloud bills
• Companies: Deploy on existing hardware
• Privacy: Keep data on-premises

The technique leverages existing llama.cpp work, but I built the Rust bindings, packaging, and curated model collection to make it actually usable for normal people.

Questions I expect

Q: Is this just quantization?
A: No, it's architectural. We're moving computation between CPU/GPU dynamically.

Q: How slow is "2-7x slower"?
A: Still interactive for most use cases. Think 10-20 tokens/sec instead of 50-100.

Q: Does this work with other models?
A: Any MoE model supported by llama.cpp. I just happen to have curated ones ready.

Q: Why not just use Ollama?
A: Ollama doesn't have MoE CPU offloading. This is the first production implementation in a user-friendly package.

Been working on this for weeks and I'm pretty excited about the implications. Happy to answer questions!

GitHub: https://github.com/Michael-A-Kuykendall/shimmy
Models: https://huggingface.co/MikeKuykendall