I created a simple implementation of the RWKV language model (RWKV competes with the dominant Transformers-based approach which is the "T" in GPT)

Repo

Right at the top for impatient people!

https://github.com/KerfuffleV2/smolrsrwkv

The repo README has a much more detailed explanation than this post.

Brief Description

This is a simple Rust implementation of RWKV. Most LLMs use Transformers (i.e. ChatGPT, etc). The creator of the RWKV approach claims it has benefits: https://github.com/BlinkDL/ChatRWKV

The purpose here is a simple implementation of RWKV (or LLM inference in general) as an aid to learning. It's not suitable for actual use, primarily because it can only use 32bit calculation. It actually runs reasonably fast (for something completely unoptimized) due to being able to run the matrix multiplication operations in parallel thanks to Rayon.

The actual math/array (tensor) handling currently uses the ndarray crate.

If you're interested in learning about RWKV, I really recommend these two links:

https://johanwind.github.io/2023/03/23/rwkv_overview.html — High level explanation.
https://johanwind.github.io/2023/03/23/rwkv_details.html — More detailed explanation with a Python example.

Example Output

The prompt given to the model is in bold. Note that it is running with a tiny
430M parameter model (generally models like this aren't too useful until around 7B or parameters).

* Loading tokenizer from: ./20B_tokenizer.json
* Loading model from: ./RWKV-4-Pile-430M-20220808-8066.safetensors
* Discovering model structure.
-   Loading layer 1/24
[...]
-   Loading layer 24/24
* Loading non-layer tensors.
* Loaded: layers=24, embed=1024, vocab=50277

In a shocking finding, scientist discovered a herd of dragons living in a remote, previously unexplored valley, in Tibet. Even more surprising to the researchers was the fact that the dragons spoke perfect Chinese.

These dragons all spoke different dialects and these dialects didn’t match the dogs' native language.

In an attempt to decipher what these dragons spoke, they called the dragons and found that their language was different from human.

"The Dragons understood human words and more precisely human languages. The dragons spoke the human language. They also understood the rules for Chinese,” the research team told Mongabay.

By conducting the research, they are hoping to shed light on the mysterious history of the dragons in the remote, remote regions of the world, especially in Tibet.

The research project, published in the journal Open Science, also shows that dragons are, in fact, reptiles, or a.k.a. tree snakes.

Dragon, not snake

According to the research team, the dragons found in Tibet are a race of dogs, not a reptile.