Update on the Candle ML framework.

We've first announced Candle, a minimalist ML framework in Rust 6 weeks ago. Since then we've focused on adding various recent models and improved the framework so as to support the necessary features in an efficient way. You can checkout a gallery of the examples, supported models include:

• Large language models: LLaMA, LLaMA v2, Falcon, Phi-v1.5, StarCoder.
• Quantized models with the llama.cpp approach: LLaMA, T5, Phi-v1.5.
• Image generation: Stable Diffusion (v1.5, v2.1, and XL), Wuerstchen.
• Computer Vision: DINOv2, yolo-v3, yolo-v8, Segment-Anything Model.
• Speech-to-text: Whisper.

One of the big upside of the pure Rust approach is that models can run directly in the browser using WASM, these can be accessed through this collection, you can try out Yolo, Whisper, Segment-Anything, T5, Llama2-c from your web browser.

Finally, in order to present a use case where Candle has unique capabilities, we've built a quantized version of the recently released Phi-v1.5 LLM. You can try it out with the following command, this uses a q4k quantized model resulting in very fast inference on CPU while still producing pretty nice texts.

$ cargo run --example phi --release -- \\
  --prompt "Explain how to find the median of an array and write the corresponding python function.\

Answer:" \
--quantized --sample-len 200

Explain how to find the median of an array and write the corresponding python function.
Answer: The median is the middle value in an array. If the array has an even number of elements, the median is the average of the two middle values.

def median(arr):
    arr.sort()
    n = len(arr)
    if n % 2 == 0:
        return (arr[n//2 - 1] + arr[n//2]) / 2
    else:
        return arr[n//2]

It's still very early days for Candle so please let us know if you start using it and run into some rough edges. We look forward to Rust getting a lot more usage in the ML space!

​

Robot holding a candle, generated by Candle Stable Diffusion XL