Rust meets Llama2: OpenAI compatible API written in Rust

Hello,

I have been working on an OpenAI-compatible API for serving LLAMA-2 models written entirely in Rust. It supports offloading computation to Nvidia GPU and Metal acceleration for GGML models thanks to the fantastic \llm\\ crate!

Here is the project link : Cria- Local LLAMA2 API

You can use it as an OpenAI replacement (check out the included \Langchain\\ example in the project).

This is an ongoing project, I have implemented the \embeddings\\ and \completions\\ routes. The \chat-completion\\ route will be here very soon!

Really interested in your feedback as I am still a beginner in Rust and I would welcome any help :) !

​

​