{"slug": "tiny-inference-engine-in-pure-rust-cpu-only-with-tui-visualization", "title": "Tiny inference engine in pure Rust, CPU-only, with TUI visualization", "summary": "A developer released Tiny-Llama, a minimal LLaMA-style inference engine for MiniCPM5-1B written in pure Rust, featuring CPU-only inference and a TUI visualization that shows how the model processes prompts in real time. The single-file implementation at src/main.rs (~1800 lines) includes core components such as RoPE, multi-head attention with GQA, SwiGLU MLP, and RMSNorm, and is designed to help others study large language model fundamentals.", "body_md": "A minimal LLaMA-style inference engine for [MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B), built purely in Rust.\n\nI built this project to study the fundamentals of large language models. Every understanding is rephrased directly in the code — dense, but organized. My hope is that it helps you learn too.\n\nInspired by [tiny-vllm](https://github.com/kuawo/tiny-llm).\n\n**CPU-only inference**— no GPU required** No KV-cache**— focuses on the core model mechanics** Single-file implementation**—`src/main.rs`\n\n(~1800 lines)**TUI visualization**— real-time view of how the model \"thinks\"\n\n| Component | Details |\n|---|---|\n| Embedding | Token ID lookup via embedding table |\n| RoPE | Rotary position embeddings |\n| Attention | Multi-head attention with GQA (Grouped Query Attention) |\n| MLP | SwiGLU activation function |\n| Normalization | RMSNorm |\n| Residual connections | Standard skip connections after attention and MLP |\n\n```\n# Download MiniCPM5-1B from HuggingFace\n# Put the model files in a directory, e.g. ./models/minicpm5-1b/\n# The directory should contain:\n#   - config.json\n#   - model-00000-of-00001.safetensors\n#   - tokenizer.json\ncargo run --release -- \"<model_dir>\" \"<your prompt>\"\n```\n\nExample:\n\n```\ncargo run --release -- ./models/minicpm5-1b \"What is artificial intelligence?\"\n```\n\nPress `q`\n\nto quit early.\n\nThe best way to read this code is top-to-bottom in `src/main.rs`\n\n:\n\n**Tensor ops**— A minimal`TinyTensor`\n\nwrapper over candle tensors with matrix multiply, reshape, transpose, softmax, etc. I will try to write the Maths operations by hand.**Model loading**—`ModelConfigurations`\n\n,`LlamaModel`\n\n, and`TransformerBlock`\n\nstructs that map directly to the safetensors keys.— The inference loop through every transformer layer. Each operation is annotated with shape comments and timing.`predict_next_token`\n\n**TUI**— Ratatui widgets render attention maps, tensor shapes, and logits in real time.\n\nMIT — see [LICENSE](/AspadaX/tiny-llama/blob/main/LICENSE)", "url": "https://wpnews.pro/news/tiny-inference-engine-in-pure-rust-cpu-only-with-tui-visualization", "canonical_source": "https://github.com/AspadaX/tiny-llama", "published_at": "2026-07-23 11:23:39+00:00", "updated_at": "2026-07-23 11:52:33.528483+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "developer-tools"], "entities": ["MiniCPM5-1B", "Rust", "Tiny-Llama", "HuggingFace", "Ratatui"], "alternates": {"html": "https://wpnews.pro/news/tiny-inference-engine-in-pure-rust-cpu-only-with-tui-visualization", "markdown": "https://wpnews.pro/news/tiny-inference-engine-in-pure-rust-cpu-only-with-tui-visualization.md", "text": "https://wpnews.pro/news/tiny-inference-engine-in-pure-rust-cpu-only-with-tui-visualization.txt", "jsonld": "https://wpnews.pro/news/tiny-inference-engine-in-pure-rust-cpu-only-with-tui-visualization.jsonld"}}