# How to Run MiniCPM5-2B Locally with SGLang or llama.cpp > Source: > Published: 2026-09-08 00:00:00+00:00 # How to Run MiniCPM5-2B Locally with SGLang or llama.cpp A practical guide to running MiniCPM5-2B locally with SGLang, covering VRAM needs, KV cache tuning, and real-world test results. ## What is MiniCPM5-2B and why does it matter? MiniCPM5-2B is a 2 billion parameter dense language model from ModelBench, a Beijing-based AI company spun out of Tsinghua University’s NLP lab. It’s the latest entry in the MiniCPM series, which has been releasing small, efficient open models for a few years. The pitch is straightforward: state of the art performance in the sub-3B parameter class, small enough to run on modest hardware but capable of agentic tool use, coding, and multi-step reasoning that used to require much larger models. Because the full weights only take up around 4GB of VRAM, it’s one of the more accessible “serious” local models to test right now, especially for anyone experimenting with agent frameworks or edge deployment. ## TL;DR - **MiniCPM5-2B** is a 2B dense model from ModelBench/Tsinghua that claims state of the art results among small open models, with benchmark comparisons against models like Qwen2.5-3B and Gemma-2B. - The model weights need roughly **4GB of VRAM** , but running it with**SGLang** and a full KV cache can push consumption up to around 44GB if you leave the cache at default settings. - You can shrink memory use dramatically by **lowering the KV cache size** in SGLang without touching the model weights themselves, making it practical on much smaller GPUs. - Training used a **three-stage pipeline** (base, mid, post-training) with a final reinforcement learning and on-policy distillation (OPD) stage that reportedly added close to 11 points on reasoning benchmarks and about 7 points on agentic tasks. - In hands-on testing, the model correctly solved a tricky **Oracle SQL partition bug** , answered a**historical fact-check question** about the Treaty of Nerchinsk accurately, and produced a surprisingly detailed**animated HTML/CSS/JS scene** of a doner kebab rotating over flames. - The model **struggled with multilingual terms of endearment** , fabricating or misstating several translations across languages, suggesting its multilingual grounding is weaker than its reasoning or coding ability. - Because it’s a reasoning model, response length can balloon fast. Running it with a **minimal reasoning effort setting** is recommended, otherwise even simple prompts take a long time to return. ## Plans first. *Then code.* Remy writes the spec, manages the build, and ships the app. ## How do you install and serve MiniCPM5-2B locally? The most direct path demonstrated is serving the model with **SGLang** on a Linux machine. SGLang handles the download automatically on first run and exposes an OpenAI-compatible API, which makes it easy to plug into existing agent frameworks or chat frontends. For anyone without a local GPU, cloud GPU rental services are a common workaround since the model doesn’t need much compute to serve. If you’d rather run on CPU or a laptop, **llama.cpp** is the usual alternative for GGUF-quantized versions of small models like this, and tools like **Ollama** or **LM Studio** typically follow once community-made quantized builds appear on Hugging Face. At the time of testing, the full-precision SGLang route was the most direct way to reproduce the benchmark-level behavior, since quantization can shift output quality, especially for a reasoning-heavy model like this one. Basic steps to serve it with SGLang: 1. Install SGLang and its dependencies on a machine with a CUDA-capable GPU. 2. Point SGLang at the MiniCPM5-2B repository ID so it downloads weights automatically. 3. Launch the server, which will bind an API endpoint you can call like any OpenAI-style chat completion endpoint. 4. Connect an agent framework (Hermes-style tool calling was used in testing) if you want to test tool use and multi-step tasks, not just chat. ## How much VRAM does MiniCPM5-2B actually need? The raw model weights take up about 4GB of VRAM, which puts it comfortably within reach of consumer GPUs. The catch is the KV cache. When served through SGLang with default settings, the KV cache allocation is maximized, which can push total VRAM usage up to roughly 44GB on a 48GB card. That number looks alarming until you realize it’s a configuration choice, not a hard requirement. Cutting the KV cache size down brings memory use back to something modest, closer to what the model weights alone need, at the cost of shorter context windows or less headroom for long agentic sessions. Practical takeaway: if you’re running on a GPU with less than 8 to 12GB of VRAM, plan to cap the KV cache explicitly rather than trusting SGLang’s default maximum allocation. ## What does MiniCPM5-2B’s training pipeline look like? The model went through a three-stage training process: base training, mid training, and post-training, with heavy investment in data quality at each stage under what ModelBench calls “ultra data.” The more interesting part is the final post-training stage. After supervised fine-tuning, the team trained separate reinforcement learning “teacher” models specialized in math, code, agentic tasks, and writing. Those specialized capabilities were then merged back into a single unified model through on-policy distillation (OPD). ### Everyone else built a construction worker. We built the contractor. One file at a time. UI, API, database, deploy. According to the reported figures, this RL-plus-OPD stage alone accounts for close to 11 points of improvement on reasoning and general benchmarks, and about 7 points on agentic task benchmarks. That’s a significant chunk of the model’s competitiveness coming from post-training rather than base pretraining scale, which lines up with a broader trend in small model development: squeezing more capability out of the same parameter count through better fine-tuning and distillation rather than just adding more data or parameters. ## Is MiniCPM5-2B actually good, or just good on paper? The benchmark comparisons published by ModelBench claim an average score of 53.9 across their evaluation suite, ahead of other small open models. But some of the comparison points raise an eyebrow: the model is stacked against older baselines like Qwen2.5-2B and Gemma-2B-instruct, plus at least one 4B parameter model, rather than the most recent competing small models. A more recent lineup of comparisons would make the claim easier to evaluate at face value. Hands-on testing tells a more mixed but largely positive story. On a hard Oracle SQL debugging task involving silent partition boundary errors, the model correctly diagnosed the bug after a long internal reasoning process, something that trips up many larger models. On a historical fact-check question about the Treaty of Nerchinsk, it answered correctly on both the language used (Latin) and the negotiation method (through Jesuit interpreters), which suggests solid factual grounding rather than pattern-matching. The standout result was a coding test: generating a self-contained HTML page simulating a rotating doner kebab over flames, complete with styling, animation, and layout details. For a 2B parameter model, the output was described as visually convincing, with working rotation, glow effects, and texture, output most people wouldn’t expect from a model this small. The weak point was multilingual translation. When asked to produce native-sounding terms of endearment across a range of languages, several answers were fabricated or incorrect, with only a handful (like Hindi and some Southeast Asian languages) landing correctly. This suggests the model’s reasoning and coding strength doesn’t fully carry over to nuanced multilingual or cultural tasks. ## Is MiniCPM5-2B worth running locally? For coding, debugging, and structured reasoning tasks, yes, it performs well above what you’d expect from a 2B model, and the low VRAM footprint makes it genuinely practical to self-host. For multilingual work or tasks requiring cultural nuance, it’s less reliable and shouldn’t be trusted without verification. Because it’s a reasoning model, response times and token usage can be much higher than a similarly sized non-reasoning model, so setting reasoning effort to minimal is advisable unless you specifically need the deeper chain-of-thought behavior for hard problems. ## Frequently Asked Questions ### How much VRAM does MiniCPM5-2B need to run? The model weights alone need about 4GB of VRAM. Total usage can climb much higher (up to around 44GB in testing) if you leave the KV cache at SGLang’s default maximum, but reducing the cache size brings memory use back down significantly. ### Can I run MiniCPM5-2B on llama.cpp or Ollama? The model can be served through llama.cpp-compatible tooling and frontends like Ollama or LM Studio once GGUF quantized versions are available, though the full-precision testing referenced here was done through SGLang. ### Why does MiniCPM5-2B take so long to respond sometimes? It’s a reasoning model that generates extended internal chains of thought before answering, sometimes recomputing the same steps multiple times. Running it at a minimal reasoning setting avoids long wait times for simple prompts. ## Seven tools to build an app. Or just Remy. Editor, preview, AI agents, deploy — all in one tab. Nothing to install. ### Is MiniCPM5-2B good at multilingual tasks? Not particularly. Testing showed it fabricated or got wrong several terms of endearment across different languages, correctly handling only a subset, which suggests weaker multilingual grounding compared to its coding and reasoning ability. ### What makes MiniCPM5-2B different from earlier MiniCPM releases? Its training pipeline includes a post-training stage that trains specialized reinforcement learning teacher models for math, code, agentic tasks, and writing, then merges their capabilities into one model via on-policy distillation, a step credited with a large share of its benchmark gains.