# Tiny inference engine in pure Rust, CPU-only, with TUI visualization

> Source: <https://github.com/AspadaX/tiny-llama>
> Published: 2026-07-23 11:23:39+00:00

A minimal LLaMA-style inference engine for [MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B), built purely in Rust.

I built this project to study the fundamentals of large language models. Every understanding is rephrased directly in the code — dense, but organized. My hope is that it helps you learn too.

Inspired by [tiny-vllm](https://github.com/kuawo/tiny-llm).

**CPU-only inference**— no GPU required** No KV-cache**— focuses on the core model mechanics** Single-file implementation**—`src/main.rs`

(~1800 lines)**TUI visualization**— real-time view of how the model "thinks"

| Component | Details |
|---|---|
| Embedding | Token ID lookup via embedding table |
| RoPE | Rotary position embeddings |
| Attention | Multi-head attention with GQA (Grouped Query Attention) |
| MLP | SwiGLU activation function |
| Normalization | RMSNorm |
| Residual connections | Standard skip connections after attention and MLP |

```
# Download MiniCPM5-1B from HuggingFace
# Put the model files in a directory, e.g. ./models/minicpm5-1b/
# The directory should contain:
#   - config.json
#   - model-00000-of-00001.safetensors
#   - tokenizer.json
cargo run --release -- "<model_dir>" "<your prompt>"
```

Example:

```
cargo run --release -- ./models/minicpm5-1b "What is artificial intelligence?"
```

Press `q`

to quit early.

The best way to read this code is top-to-bottom in `src/main.rs`

:

**Tensor ops**— A minimal`TinyTensor`

wrapper over candle tensors with matrix multiply, reshape, transpose, softmax, etc. I will try to write the Maths operations by hand.**Model loading**—`ModelConfigurations`

,`LlamaModel`

, and`TransformerBlock`

structs that map directly to the safetensors keys.— The inference loop through every transformer layer. Each operation is annotated with shape comments and timing.`predict_next_token`

**TUI**— Ratatui widgets render attention maps, tensor shapes, and logits in real time.

MIT — see [LICENSE](/AspadaX/tiny-llama/blob/main/LICENSE)