{"slug": "old-xeons-can-run-gemma-4-at-reading-speed", "title": "Old Xeons Can Run Gemma 4 at Reading Speed", "summary": "A 26-billion-parameter Gemma 4 mixture-of-experts model can run at reading speed on decade-old Xeon CPUs, achieving ~5.2 tokens per second on a dual-socket 2013 Ivy Bridge system with no GPU, thanks to MoE architecture and optimized CPU kernels. The finding shows that CPU inference for modern MoE models has become a viable option for local deployment on low-cost hardware.", "body_md": "[AI](https://sourcefeed.dev/c/ai)Article\n\n# Old Xeons Can Run Gemma 4 at Reading Speed\n\nMoE math, CPU kernels, and speculative decoding turn retired enterprise boxes into real local inference hosts.\n\n[Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein)\n\nAnybody can rent a GPU. The more interesting question is what happens when a 26-billion-parameter open model has to earn its keep on hardware that was obsolete before the architecture existed.\n\n[Gemma 4](https://ai.google.dev/gemma)'s 26B-A4B is a mixture-of-experts model: roughly 25.2B total parameters, about 3.8B active per token (8 of 128 experts plus a shared path). That asymmetry is not a marketing footnote. It is the reason a dual-socket 2013 HP StoreVirtual full of Ivy Bridge Xeons, DDR3, and zero GPUs can decode at ~5.2 tokens per second on a Q8_0 quant, with prompt eval around 16 tok/s. Reading speed. On a box that cost under $300 used and was built to hold disks, not do matmuls.\n\nThe thesis is simple. CPU inference for modern MoE models has crossed from stunt into option. The remaining hard parts are not \"can it run\" but instruction-set floors, which fork you trust, and whether reading-speed local generation is the right trade for the job.\n\n## Why MoE rewrites the CPU budget\n\nDecode is memory-bound. Every generated token has to haul weights from RAM into cache while the cores wait. On a GPU you buy HBM and call it a day. On a Xeon you live inside L3 and the memory wall.\n\nDense 26B models on CPU are a slideshow for that reason. The 26B-A4B is different because most of the weight mass stays cold. Only the routed experts move. Community reports put the official QAT Q4 around 14.4GB on disk, which is why a 32GB mini PC or used desktop becomes a realistic host and why denser 12B and 31B variants still fight for RAM and bandwidth the MoE side-steps.\n\nThat is also why a mid-tier 2018-era i5-8500 with 32GB and no GPU can land ~7 tok/s on Q4. Same model family, different quant and clock, same lesson: active parameter count dominates wall-clock on CPU far more than the \"26B\" label on the file.\n\nBenchmarks still matter. One community framing put Gemma 4 26B-A4B near 82.6% on MMLU Pro. Quality like a mid-size dense model with the active cost of a small one is the entire product. If you only remember one architectural fact for local deployment this year, remember that.\n\n## What \"usable\" looks like on junk silicon\n\nTwo independent setups make the floor concrete.\n\nA recycled 2016 Broadwell Xeon E5-2620 v4 with 128GB of slow DDR3, no GPU, AVX2 but nothing fancier, runs the same 26B-A4B once you stop using stock [llama.cpp](https://github.com/ggerganov/llama.cpp) as if it were a GPU runtime. The working configuration leans on ik_llama.cpp: multi-token prediction style speculative decoding against a small drafter, CPU-aware MoE routing, flash attention on CPU, runtime weight repacking, merged up/gate expert paths, and a long list of graph and parallel flags that generic wrappers never expose.\n\nA 2013 dual E5-2690 v2 box is older still. Ivy Bridge has AVX1 only. No AVX2, no FMA3. The same optimized fork assumes AVX2 as its floor, so the hot IQK matmul paths do not simply slow down. Two MoE fused ops (`MOE_FUSED_UP_GATE`\n\nand `FUSED_UP_GATE`\n\n) still get emitted by the graph builder, miss their dispatcher cases when the fast kernels are compiled out, and leave expert FFN destinations uncomputed. Symptom: deterministic multilingual gibberish with token IDs smeared across a 262K vocabulary. Thirty layers times eight active experts is on the order of 240 tensors of whatever garbage already lived in those buffers.\n\nThe fix is not \"turn the flag off and hope.\" It is fallback kernels that keep the fused MoE path valid on pre-AVX2 parts. With that in place, Q8_0 decode lands near 5.2 tok/s on the dual-socket Ivy Bridge machine. Prompt eval is faster than decode, as usual, because prefill is more compute-friendly than the serial memory crawl of generation.\n\nFive tokens per second is not a coding autocomplete firehose. It is fine for offline chat, batch summarization when you are not paying per token, and a local fallback when the SaaS endpoints are down or blocked. That is the actual product category these boxes fall into.\n\n## The developer path: forks, floors, and trade-offs\n\nIf you already own a 32GB+ machine with a vaguely modern x86 CPU, the practical stack is not Ollama-with-defaults. Start from a fork that actually ships the MoE and CPU work (ik_llama.cpp is the one these runs depend on), grab Google's QAT GGUFs when you care about quality-per-byte, and treat the flag list as load-bearing rather than cargo cult.\n\nRough decision tree:\n\n**RAM.** Q4-class 26B-A4B wants ~32GB to feel like a daily driver, not a science project. 16GB is the 12B dense lane. Sub-16GB is E4B/E2B territory.**Quant.** Q8_0 on ancient dual-socket iron buys headroom in quality and still fits when you have enterprise DDR3 by the bucket. Q4 QAT is the right default on 32GB desktops and mini PCs.**ISA floor.** AVX2 is the real baseline for the fast kernels. Pre-AVX2 (Ivy Bridge and friends) needs explicit fallbacks or you get fluent garbage, not a clean slow path. Check`lscpu`\n\nbefore you blame the model.**Speculative decoding.** Stronger argument on CPU than GPU: the drafter's working set can sit in L3 while the verifier is bandwidth-bound. Worth the complexity for interactive use; less critical for overnight batch.**What it replaces.** Not an H100. Not a 70B dense chat box. It replaces \"I have no offline path\" and \"this batch job is not worth API spend.\"\n\nTrade-offs are honest. You will spend time on build flags and graph ops instead of a one-click app. Throughput stays in the single-digit tok/s band on old DDR3. Context and concurrent users eat the same RAM the weights need. And MoE routing that thrashes cache without `--cpu-moe`\n\n-style care will throw away the architectural advantage you paid for in model choice.\n\nIf the hardware is already paid for, the marginal cost is electricity and your patience. If you are buying, a used 32GB desktop still beats a new GPU for this specific 26B-A4B profile unless you need interactive multi-user speed.\n\n## Who wins\n\nLocal MoE on CPU is no longer a weekend dare. The combination of sparse activation, QAT quants, speculative decoding, and CPU-specific kernels has made \"retired enterprise junk\" a legitimate tier for open weights that would have been datacenter-only a couple of model generations ago.\n\nWho wins: people sitting on dual-socket leftovers, labs that want an air-gapped fallback, and anyone grinding slow batch jobs where tokens-per-dollar matter more than tokens-per-second. Who should ignore this: teams that need multi-user latency, tool-heavy agents that thrash long contexts, or anyone who cannot maintain a non-default llama.cpp fork.\n\nThe skill that ages better than a subscription is knowing which model, quant, and kernel path fit the iron you already have, and being able to tell when the answer is correct. Five tokens a second on a 13-year-old Xeon is not a flex about nostalgia. It is evidence the optimization stack finally caught up to the hardware people already own.\n\n## Sources & further reading\n\n-\n[Running Gemma 4 26B at 5 tokens/SEC on a 13-year-old Xeon with no GPU](https://www.neomindlabs.com/2026/06/08/running-gemma-4-26b-at-5-tokens-sec-on-a-13-year-old-xeon-with-no-gpu/)— neomindlabs.com -\n[A 10 year old Xeon is all you need - point.free](https://point.free/blog/gemma-4-on-a-2016-xeon/)— point.free -\n[Run Gemma 4 on a mini PC, no GPU required | TerminalBytes](https://terminalbytes.com/run-gemma-4-mini-pc-without-gpu/)— terminalbytes.com -\n[Gemma 4 Requirements: Mac & GPU Guide (All 5 Models) | Gemma4All](https://gemma4all.com/blog/gemma-4-hardware-requirements)— gemma4all.com -\n[Gemma 4 - How to Run Locally | Unsloth Documentation](https://unsloth.ai/docs/models/gemma-4)— unsloth.ai\n\n[Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein)· Dev Tools Editor\n\nRachel has been embedded in the developer tooling ecosystem for nearly eight years, covering everything from IDE wars and package-manager drama to the quiet rise of AI-assisted coding. She has a soft spot for open-source maintainers and an unhealthy number of terminal emulators installed on a single laptop.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/old-xeons-can-run-gemma-4-at-reading-speed", "canonical_source": "https://sourcefeed.dev/a/old-xeons-can-run-gemma-4-at-reading-speed", "published_at": "2026-07-15 17:03:55+00:00", "updated_at": "2026-07-15 17:04:50.438300+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-infrastructure", "ai-tools"], "entities": ["Google", "Gemma 4", "llama.cpp", "HP", "Ivy Bridge", "Broadwell", "Xeon E5-2620 v4", "Xeon E5-2690 v2"], "alternates": {"html": "https://wpnews.pro/news/old-xeons-can-run-gemma-4-at-reading-speed", "markdown": "https://wpnews.pro/news/old-xeons-can-run-gemma-4-at-reading-speed.md", "text": "https://wpnews.pro/news/old-xeons-can-run-gemma-4-at-reading-speed.txt", "jsonld": "https://wpnews.pro/news/old-xeons-can-run-gemma-4-at-reading-speed.jsonld"}}