Tiny inference engine in pure Rust, CPU-only, with TUI visualization

A developer released Tiny-Llama, a minimal LLaMA-style inference engine for MiniCPM5-1B written in pure Rust, featuring CPU-only inference and a TUI visualization that shows how the model processes prompts in real time. The single-file implementation at src/main.rs (~1800 lines) includes core components such as RoPE, multi-head attention with GQA, SwiGLU MLP, and RMSNorm, and is designed to help others study large language model fundamentals.

A minimal LLaMA-style inference engine for MiniCPM5-1B https://huggingface.co/openbmb/MiniCPM5-1B , built purely in Rust. I built this project to study the fundamentals of large language models. Every understanding is rephrased directly in the code — dense, but organized. My hope is that it helps you learn too. Inspired by tiny-vllm https://github.com/kuawo/tiny-llm . CPU-only inference — no GPU required No KV-cache — focuses on the core model mechanics Single-file implementation — src/main.rs ~1800 lines TUI visualization — real-time view of how the model "thinks" | Component | Details | |---|---| | Embedding | Token ID lookup via embedding table | | RoPE | Rotary position embeddings | | Attention | Multi-head attention with GQA Grouped Query Attention | | MLP | SwiGLU activation function | | Normalization | RMSNorm | | Residual connections | Standard skip connections after attention and MLP | Download MiniCPM5-1B from HuggingFace Put the model files in a directory, e.g. ./models/minicpm5-1b/ The directory should contain: - config.json - model-00000-of-00001.safetensors - tokenizer.json cargo run --release -- "<model dir " "<your prompt " Example: cargo run --release -- ./models/minicpm5-1b "What is artificial intelligence?" Press q to quit early. The best way to read this code is top-to-bottom in src/main.rs : Tensor ops — A minimal TinyTensor wrapper over candle tensors with matrix multiply, reshape, transpose, softmax, etc. I will try to write the Maths operations by hand. Model loading — ModelConfigurations , LlamaModel , and TransformerBlock structs that map directly to the safetensors keys.— The inference loop through every transformer layer. Each operation is annotated with shape comments and timing. predict next token TUI — Ratatui widgets render attention maps, tensor shapes, and logits in real time. MIT — see LICENSE /AspadaX/tiny-llama/blob/main/LICENSE