Early preview: Candle - torch in Rust

Hey there:

For the past month or so Laurent Mazare (creator of tch-rs) has been working on a replacement for torch in Rust: https://github.com/huggingface/candle/

This is still very early on but there is already:

\- Llama 1&2, Whisper, Bert, StarCoder, Falcon support

\- Works on GPU, CPU, M1, WASM- Works with flash attention v2

In terms of performance it´'s still rough around in the edges but :- we can basically get the same latency as text-generation-inference in TP=4 for llamav2 (21ms/token vs 19ms/token) and there are still quite a few optimizations to make.

\- For CPU you will get the best performance using \mkl\\ right now. We´'re working on improving the \gemm\\ crate to provide support for all platforms with as close as possible to the best speed. (Outstanding issue: https://github.com/huggingface/candle/pull/323)

Wasm demos: https://huggingface.co/spaces/lmz/candle-whisper https://huggingface.co/spaces/lmz/candle-llama2

we ve also started a book: https://huggingface.github.io/candle/ where we will update a bit the best practices like running a model with tracing to get live performance.

We wrote that initially for much smaller binary sizes and getting down to serverless, but it is very practical and we hope we can push the performance boundary with it against both torch and GGML. For GGML we don´t necessarily aim to beat it on every single benchmark and whatnot, but being close enough in performance, and closer to existing pytorch code so we could leverage existing modeling code and models out there.

Stable diffusion is coming soon :https://github.com/huggingface/candle/pull/342

We also have Q4 (ggml + GPTQ versions) in mind.Lots of cleanup to do and improving the overall DX.

This work has been sponsored by huggingface, and the license is and will remain Apache 2.0 + MIT

Contributions are welcome, but please by mindful that it is still quite early on, the code hasn´t been polished yet.

We re also trying to merge as much code as possible upstream so that other projects could benefit https://github.com/sarah-ek/gemm https://github.com/starkat99/half-rs https://github.com/coreylowman/cudarc