A minimal LLaMA-style inference engine for MiniCPM5-1B, built purely in Rust.
I built this project to study the fundamentals of large language models. Every understanding is rephrased directly in the code — dense, but organized. My hope is that it helps you learn too.
Inspired by tiny-vllm.
CPU-only inference— no GPU required** No KV-cache**— focuses on the core model mechanics** Single-file implementation**—src/main.rs
(~1800 lines)TUI visualization— real-time view of how the model "thinks"
| Component | Details |
|---|---|
| Embedding | Token ID lookup via embedding table |
| RoPE | Rotary position embeddings |
| Attention | Multi-head attention with GQA (Grouped Query Attention) |
| MLP | SwiGLU activation function |
| Normalization | RMSNorm |
| Residual connections | Standard skip connections after attention and MLP |
cargo run --release -- "<model_dir>" "<your prompt>"
Example:
cargo run --release -- ./models/minicpm5-1b "What is artificial intelligence?"
Press q
to quit early.
The best way to read this code is top-to-bottom in src/main.rs
:
Tensor ops— A minimalTinyTensor
wrapper over candle tensors with matrix multiply, reshape, transpose, softmax, etc. I will try to write the Maths operations by hand.**Model **—ModelConfigurations
,LlamaModel
, andTransformerBlock
structs that map directly to the safetensors keys.— The inference loop through every transformer layer. Each operation is annotated with shape comments and timing.predict_next_token
TUI— Ratatui widgets render attention maps, tensor shapes, and logits in real time.
MIT — see LICENSE