Running Gemma 4 26B at 5 tokens/SEC on a 13-year-old Xeon with no GPU A developer running Google's Gemma 4 26B mixture-of-experts model on a 13-year-old HP StoreVirtual server with dual Ivy Bridge Xeons and no GPU achieved 5 tokens per second by patching inference kernels to work without AVX2 and FMA3 instruction sets, demonstrating that modern AI can run on obsolete hardware with careful engineering. June 8, 2026 · By Ryan Findley Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU There’s a server in my basement that has no business running a modern language model. It’s a repurposed HP StoreVirtual storage box, roughly thirteen years old, two Ivy Bridge Xeons, no GPU. It was built to hold disks, not do math. As of this week it runs Google’s Gemma 4 https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/ , a 26-billion-parameter open-weights mixture-of-experts model, at about five tokens per second. Reading speed. | Hardware | Repurposed HP StoreVirtual: dual Xeon E5-2690 v2 Ivy Bridge, 2013 , DDR3, no GPU | | Instruction sets | AVX1 only — no AVX2, no FMA3 | | Model | Gemma 4 26B-A4B MoE , Q8 0 | | Decode | ~5.2 tokens/sec | | Prompt eval | ~16 tokens/sec | | Cost of the box | under $300 | Anybody can rent a GPU. It’s harder to take a modern MoE model and a dead enterprise box and make them meet in the middle, and that gap is the whole reason I’m writing this up. “Good with AI” has quietly come to mean “pays for a subscription.” I think the real skill is different: knowing a model well enough to point it at a problem nobody packaged for you, and telling whether the answer it hands back is actually correct. So rather than claim we’re good at this, here’s a worked example, on hardware that had no business cooperating. The post that started it A couple of weeks ago a piece called “A 10 year old Xeon is all you need” https://point.free/blog/gemma-4-on-a-2016-xeon/ made the rounds on Hacker News. The author runs Gemma 4 on a single 2016 Xeon with no GPU and 128 GB of slow DDR3, using ik llama.cpp https://github.com/ikawrakow/ik llama.cpp and about 25 carefully chosen flags. It’s a great read, and it leans on every trick in the modern inference playbook: speculative decoding, CPU-aware mixture-of-experts routing, flash attention ported to the CPU, run-time weight repacking. Real engineering. “I have a Xeon too,” I thought. Several, in fact. So I tried it. It didn’t run. What an AI agent is actually good for The build died on startup. I handed the failure to Claude and asked what was wrong. The answer came back fast and specific. The author’s 2016 chip is a Broadwell part. Mine are Ivy Bridge, the generation Intel calls “v2.” The fast kernels in that fork assume AVX2 and FMA3, instruction sets that didn’t ship until Haswell, the “v3” generation, in 2014 https://www.microway.com/hpc-tech-tips/avx2-optimization-and-haswell-ep-cpu-features/ . My CPUs are older than the instructions the code was written against. The optimized paths weren’t there to execute. So I asked the obvious follow-up: can we make it run anyway? I’d already taken a first swing with a free model that got close but couldn’t land it. Claude picked up that half-finished approach, agreed it was the right one, and finished it off, reworking the hot paths so they fall back cleanly on a pre-AVX2 chip instead of reaching for instructions that aren’t there. This is the part I care about. This didn’t come from typing “fix it” once and getting a working patch back. Somebody had to read another person’s performance-critical C++, work out why a kernel wasn’t valid on this particular microarchitecture, and route around it without throwing away the optimizations that made the fork worth using. Claude did that work. My job was narrower: run the right experiments and recognize when the output was finally correct. I came away impressed. The result Gemma 4’s 26B mixture-of-experts model now generates text at reading speed on hardware that was retired before the model’s architecture existed. The original write-up never published a tokens-per-second figure, just “reading speed,” so here’s the concrete one: about five tokens a second on thirteen-year-old silicon, for borderline free. Proof it runs: Gemma 4 26B answering on the basement box, CPU-only. The patch is up as ikawrakow/ik llama.cpp 2138 https://github.com/ikawrakow/ik llama.cpp/pull/2138 if you want the exact diff — still open and awaiting maintainer review as I write this, so run it from the branch for now. The hope is that anyone else sitting on ancient enterprise iron can keep a local model around: a fallback for when the paid APIs are down, or a cheap way to grind through slow batch jobs when paying per token doesn’t make sense. For the people who want the actual bug Full disclosure before I go further. I’m not a C++ programmer. I can read a stack trace and I know my way around a build system, but I did not hand-write kernel fallbacks for a quantized matmul engine, and I won’t pretend I did. What I did was drive. I ran the experiments, read the output, asked the next question, and knew what “correct” had to look like. The diagnosis and the patch came from the Claude instance running on the server itself. I asked it to write up what it fixed, and the rest of this section is that summary, lightly edited. If you came here from Hacker News for the real teardown, this part’s for you. What was actually broken The engine we needed was ik llama.cpp , ikawrakow’s fork of llama.cpp that adds the optimizations Gemma 4’s MoE inference depends on. It assumes AVX2 as its floor. The Xeon E5-2690 v2 in this box has AVX1 but not AVX2. Turn GGML USE IQK MULMAT off at build time and most of the codebase respects it: the fast paths compile out, and the model falls back to plain scalar/SSE math. That’s fine for a normal Q8 0 matmul. Two graph ops are the exception. The Gemma 4 MoE feed-forward network emits MOE FUSED UP GATE a per-expert gate+up matmul fused with SwiGLU and FUSED UP GATE its dense analog . Both are if -gated on GGML USE IQK MULMAT inside the compute dispatcher, but the graph builder still emits them unconditionally. On this build the dispatcher’s switch had no case for those op enums, so they fell through to the default, and the destination tensors for every expert FFN silently never got computed. Gemma 4 26B has 30 layers by 8 active experts per token, so every forward pass consumed roughly 240 tensors of whatever happened to be sitting in that memory buffer already. The symptom was fluent-looking multilingual gibberish. Token IDs spread uniformly across the 262K vocabulary, the model equally happy to emit Thai script, Korean,