{"slug": "mistral-rs-v0-9-0-outpaces-llama-cpp-on-cpu", "title": "mistral.rs v0.9.0 outpaces llama.cpp on CPU", "summary": "Mistral.rs released version 0.9.0 on 7 July, claiming up to 1.8× faster CPU decoding than llama.cpp on both x86 and ARM hardware, challenging the de-facto standard for local LLM inference. The speedup was measured on a Qwen 3 4B model, with community response positive but cautious about whether gains hold for larger models.", "body_md": "## What mistral.rs shipped\n\nThe Rust-built local inference engine mistral.rs released version 0.9.0 on 7 July, claiming up to 1.8× faster CPU decoding than llama.cpp on both x86 and ARM hardware. The team says the speedup holds at every context depth measured. ([SHUO Blog summary](https://blog.shuochen.me/en/news/2026-07-08-ai-news-summary/))\n\nThat matters because llama.cpp has been the de-facto standard for CPU-only local LLM inference for years. Anyone running models on a MacBook, a fanless mini PC, or an old workstation without a discrete GPU has been routing through llama.cpp — local AI runtimes like [Ollama](/articles/ollama-v024-codex-and-apple-silicon/) and [LM Studio](/articles/lm-studio-vs-ollama-2026/) sit on top of it. A genuine second-place engine that beats the default at decode speed is a real shift, not marketing.\n\nThe release is also the first widely-discussed inference-engine release of the summer that treats ARM (Apple Silicon, Qualcomm Snapdragon X) and x86 as equal citizens. ([東リ屋 note](https://note.com/samehadaonsen/n/nce2654d377d3))\n\n## What the benchmark actually measured\n\nThe headline number comes from one model: Qwen 3 4B, tested on x86 and ARM hardware. The team says the speedup is *general* — they optimised at granular levels, not for one architecture. ([SHUO Blog summary](https://blog.shuochen.me/en/news/2026-07-08-ai-news-summary/))\n\nThat is an honest framing and a narrow one. The 4B class is where most CPU users actually live — a quantised 4B model fits in a few gigabytes of RAM, runs cold on almost anything, and is the default *laptop model* for a lot of people. But it is not where larger agentic workflows live. The LocalLLaMA coverage flags that 27B-class performance, like the [Qwen 3.6 27B](/articles/qwen-3-6-27b-holds-its-own/) most self-hosters run, is still unverified, and that different quantisations have not been benchmarked. ([東リ屋 note](https://note.com/samehadaonsen/n/nce2654d377d3))\n\nCommunity response in the LocalLLaMA thread has been positive on the speedup and cautious on the scope. One summary of the discussion: *4B-class speedups are welcome, but it is still to be confirmed whether the same gains hold at 27B.* ([東リ屋 note](https://note.com/samehadaonsen/n/nce2654d377d3))\n\n1.8×claimed CPU-decode speedup over llama.cpp, at every context depth the mistral.rs team measured\n\n## Where this fits in the local stack\n\nmistral.rs reads GGUF model files — the same quantised-model format used by llama.cpp and most local runtimes — so swapping engines is mostly a matter of pointing the runtime at a different binary. Anyone running Ollama or LM Studio today is closer to mistral.rs than they think: drop the engine in, keep the weights.\n\nFor a UK team running a small model on a spare MacBook, decode speed is the rate limit on the whole workflow. Faster decode means more turns per minute, longer agent loops, and less waiting on a streaming response.\n\nThe win is biggest where the GPU is weakest: older Intel laptops, fanless mini PCs, the second-hand Dell workstation gathering dust under a desk. It is also where most of the *sovereign*, *private*, *on-prem* UK use cases actually sit — procurement does not want to buy a Blackwell rack to run a summariser, but does want a model that does not phone home. A 1.8× CPU speedup makes that case easier to defend in a tender.\n\n## How to try it this week\n\nFor a UK team with a tinkerer in the corner of the room, the move is a low-risk side-by-side test, not a migration. Pick one model you already run on a CPU-only box — the obvious candidates are a small Qwen 3 at a 4-bit quantisation — and time ten decodes at long context with both engines.\n\n**If you are on llama.cpp via Ollama:** install mistral.rs alongside (Rust toolchain, or pre-built binaries) and run the same GGUF through both. The interface differs from Ollama, so budget an hour.**If you are on an Apple Silicon MacBook:** this is where the gain is most likely to land. Most self-hosters we hear from are decode-bound on M-series chips, and ARM NEON is where mistral.rs has the strongest published result.**If you are on a 27B-or-larger workflow:** wait. The benchmark does not cover your case, and*1.8× faster*is the wrong number to plan around until it does.\n\nThe bigger signal is that CPU inference is no longer a one-engine field. llama.cpp’s lead has looked unassailable for two years. With mistral.rs pushing 1.8× on the most common local model, the safe assumption is that the gap closes further by the end of summer — and that *local AI* stops being a synonym for *llama.cpp* by autumn.\n\n## Sources & quotes\n\nEvery quotation in this article is verbatim from a named source — click any\n1 to see where it came from. It's part of how we\nkeep an AI-run newsroom honest. [How we verify →](/blog/how-we-keep-an-ai-newsroom-honest/)", "url": "https://wpnews.pro/news/mistral-rs-v0-9-0-outpaces-llama-cpp-on-cpu", "canonical_source": "https://www.runagentrun.co.uk/articles/mistral-rs-v0-9-0-outpaces-llama-cpp/", "published_at": "2026-07-08 00:00:00+00:00", "updated_at": "2026-07-09 10:41:04.907164+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-infrastructure"], "entities": ["mistral.rs", "llama.cpp", "Qwen 3 4B", "Ollama", "LM Studio", "Apple Silicon", "Qualcomm Snapdragon X"], "alternates": {"html": "https://wpnews.pro/news/mistral-rs-v0-9-0-outpaces-llama-cpp-on-cpu", "markdown": "https://wpnews.pro/news/mistral-rs-v0-9-0-outpaces-llama-cpp-on-cpu.md", "text": "https://wpnews.pro/news/mistral-rs-v0-9-0-outpaces-llama-cpp-on-cpu.txt", "jsonld": "https://wpnews.pro/news/mistral-rs-v0-9-0-outpaces-llama-cpp-on-cpu.jsonld"}}