A thing to run big models across multiple machines over WiFi

Some of you may remember me from corroded. Since then everyone thinks I'm a troll and I get angry executive messages on LinkedIn. Decided to work on something more useful this time.

I had a few macbooks lying around and thought maybe I can split a model across these and run inference. Turns out I can.

I split the model across machines and runs inference as a pipeline. Works over WiFi. You can mix silicon, nvidia, cpu, whatever.

Theoretically your smart fridge and TV could join the cluster. I haven't tried this, yet. I don't have enough smart fridges.

Repo is here.

Disclaimer: I haven't tested a 70B model because I don't have the download bandwidth. I'm poor. I need to go to the office just to download the weights. I'll do that eventually. Been testing with tinyllama and it works great.

PS: I'm aware of exo and petals.