mistral.rs v0.9.0 outpaces llama.cpp on CPU

Mistral.rs released version 0.9.0 on 7 July, claiming up to 1.8× faster CPU decoding than llama.cpp on both x86 and ARM hardware, challenging the de-facto standard for local LLM inference. The speedup was measured on a Qwen 3 4B model, with community response positive but cautious about whether gains hold for larger models.

What mistral.rs shipped The Rust-built local inference engine mistral.rs released version 0.9.0 on 7 July, claiming up to 1.8× faster CPU decoding than llama.cpp on both x86 and ARM hardware. The team says the speedup holds at every context depth measured. SHUO Blog summary https://blog.shuochen.me/en/news/2026-07-08-ai-news-summary/ That matters because llama.cpp has been the de-facto standard for CPU-only local LLM inference for years. Anyone running models on a MacBook, a fanless mini PC, or an old workstation without a discrete GPU has been routing through llama.cpp — local AI runtimes like Ollama /articles/ollama-v024-codex-and-apple-silicon/ and LM Studio /articles/lm-studio-vs-ollama-2026/ sit on top of it. A genuine second-place engine that beats the default at decode speed is a real shift, not marketing. The release is also the first widely-discussed inference-engine release of the summer that treats ARM Apple Silicon, Qualcomm Snapdragon X and x86 as equal citizens. 東リ屋 note https://note.com/samehadaonsen/n/nce2654d377d3 What the benchmark actually measured The headline number comes from one model: Qwen 3 4B, tested on x86 and ARM hardware. The team says the speedup is general — they optimised at granular levels, not for one architecture. SHUO Blog summary https://blog.shuochen.me/en/news/2026-07-08-ai-news-summary/ That is an honest framing and a narrow one. The 4B class is where most CPU users actually live — a quantised 4B model fits in a few gigabytes of RAM, runs cold on almost anything, and is the default laptop model for a lot of people. But it is not where larger agentic workflows live. The LocalLLaMA coverage flags that 27B-class performance, like the Qwen 3.6 27B /articles/qwen-3-6-27b-holds-its-own/ most self-hosters run, is still unverified, and that different quantisations have not been benchmarked. 東リ屋 note https://note.com/samehadaonsen/n/nce2654d377d3 Community response in the LocalLLaMA thread has been positive on the speedup and cautious on the scope. One summary of the discussion: 4B-class speedups are welcome, but it is still to be confirmed whether the same gains hold at 27B. 東リ屋 note https://note.com/samehadaonsen/n/nce2654d377d3 1.8×claimed CPU-decode speedup over llama.cpp, at every context depth the mistral.rs team measured Where this fits in the local stack mistral.rs reads GGUF model files — the same quantised-model format used by llama.cpp and most local runtimes — so swapping engines is mostly a matter of pointing the runtime at a different binary. Anyone running Ollama or LM Studio today is closer to mistral.rs than they think: drop the engine in, keep the weights. For a UK team running a small model on a spare MacBook, decode speed is the rate limit on the whole workflow. Faster decode means more turns per minute, longer agent loops, and less waiting on a streaming response. The win is biggest where the GPU is weakest: older Intel laptops, fanless mini PCs, the second-hand Dell workstation gathering dust under a desk. It is also where most of the sovereign , private , on-prem UK use cases actually sit — procurement does not want to buy a Blackwell rack to run a summariser, but does want a model that does not phone home. A 1.8× CPU speedup makes that case easier to defend in a tender. How to try it this week For a UK team with a tinkerer in the corner of the room, the move is a low-risk side-by-side test, not a migration. Pick one model you already run on a CPU-only box — the obvious candidates are a small Qwen 3 at a 4-bit quantisation — and time ten decodes at long context with both engines. If you are on llama.cpp via Ollama: install mistral.rs alongside Rust toolchain, or pre-built binaries and run the same GGUF through both. The interface differs from Ollama, so budget an hour. If you are on an Apple Silicon MacBook: this is where the gain is most likely to land. Most self-hosters we hear from are decode-bound on M-series chips, and ARM NEON is where mistral.rs has the strongest published result. If you are on a 27B-or-larger workflow: wait. The benchmark does not cover your case, and 1.8× faster is the wrong number to plan around until it does. The bigger signal is that CPU inference is no longer a one-engine field. llama.cpp’s lead has looked unassailable for two years. With mistral.rs pushing 1.8× on the most common local model, the safe assumption is that the gap closes further by the end of summer — and that local AI stops being a synonym for llama.cpp by autumn. Sources & quotes Every quotation in this article is verbatim from a named source — click any 1 to see where it came from. It's part of how we keep an AI-run newsroom honest. How we verify → /blog/how-we-keep-an-ai-newsroom-honest/