Ollama 0.31 ships a multi-token prediction update for Gemma 4 on Apple Silicon that cuts token generation time by nearly 90% on a real coding benchmark. No configuration. No quality tradeoff. Pull the update, grab the MLX model variant, and the speedup kicks in automatically. For developers running coding agents locally — Aider, opencode, or anything that chains sequential LLM calls — this changes the latency math in a way that matters.
The 90% Number Is From a Real Coding Benchmark #
The benchmark is Aider’s polyglot suite: 225 of the hardest coding exercises from Exercism, spread across C++, Go, Java, JavaScript, Python, and Rust. It tests autonomous AI editing across your actual tech stack, not cherry-picked Python puzzles. Ollama tested Gemma 4 12B in its nvfp4 quantization on an Apple Silicon M5 Max and measured token generation speed before and after multi-token prediction. The result was 90% faster on average across that benchmark.
That number is high because code is more predictable than prose. A draft model guessing the next few tokens in a for loop or function signature is right far more often than a model guessing the next word in a paragraph. Higher acceptance rate means more tokens committed per verification pass, which means the speedup compounds. Code gets an outsized benefit relative to general text.
How Multi-Token Prediction Actually Works #
Standard LLM inference is memory-bandwidth bound: the model produces one token per forward pass, spending the same compute on an obvious closing bracket as on a complex reasoning step. Multi-token prediction breaks that bottleneck with a draft model — a small, fast companion trained alongside the main Gemma 4 model — that proposes several tokens at once. The main model then verifies all proposals in a single parallel pass, accepts the longest correct prefix, and appends one correction token of its own. Rejected tokens are rolled back via cached state, not recomputed. The output is mathematically identical to standard greedy decoding. There is no quality tradeoff — only throughput changes.
Ollama adds an auto-tuning layer on top: it monitors the acceptance rate and verification time in real-time, adjusting how many tokens to draft to maximize tokens per second. When speculation starts costing more than it saves, the system backs off automatically. No knob to turn, no edge case to manage.
How to Get It #
You need Ollama 0.31 or 0.31.1 and the MLX variant of Gemma 4. Three commands:
brew upgrade ollama
curl -fsSL https://ollama.com/install.sh | sh
ollama pull gemma4:12b-mlx
ollama run gemma4:12b-mlx
Requirements: Apple Silicon Mac (M1 through M5), macOS, at least 16GB unified memory for the 12B model. The MLX backend is selected automatically on Apple Silicon. The model name suffix matters: gemma4:12b-mlx
, not gemma4:12b
. Pulling the non-MLX variant gives you the standard llama.cpp path without MTP.
Where You Will Actually Feel It #
Agentic loops are where the improvement is most tangible. A single fast turn is nice. Five consecutive turns is a different experience, because latency compounds across the chain. The table below uses a conservative 50 tok/s baseline (typical for Gemma 4 12B without MTP on Apple Silicon) and a post-MTP rate of roughly 90 tok/s:
| Scenario | Without MTP (~50 tok/s) | With MTP (~90 tok/s) |
|---|---|---|
| 1,000-token agent turn | 20 sec | 11 sec |
| 3,000-token code review | 60 sec | 33 sec |
| 5-turn agent loop (1k tokens each) | 100 sec | 56 sec |
The delta between 100 seconds and 56 seconds is not an abstract performance metric. It is the difference between a tool that breaks your flow and one that maintains it. If you use Aider or opencode as a primary coding assistant on a Mac, upgrading now is worth the two minutes it takes.
What Is Coming Next #
Ollama says “more models to come” for MTP support. Gemma 4 is first because it ships with dedicated draft heads trained alongside the main model — not all models have this. Qwen 3 and Llama 4 are natural next candidates given their Ollama install counts, but Ollama has not confirmed a timeline.
This optimization is currently Apple Silicon and MLX only. Developers on Linux or Windows with CUDA do not get it in 0.31. The speculative decoding mechanism works on GPU hardware — this is an implementation priority question, not a fundamental limitation.
The broader point: local inference is getting harder to dismiss on performance grounds. Gemma 4 12B running at 90 tok/s on a Mac you already own, for free, with your code never leaving the machine, is a credible alternative to a paid cloud API for most coding agent workloads. That statement would have been a stretch eighteen months ago. It is not a stretch now.