Decode Speed Lies: phi4:14b Loses 79.7% of Its Throughput Before You See a Token

An analysis of 2,218 instrumented production calls over 30 days found that phi4:14b delivered only 25.3 tokens per second to callers versus its 124.7 tok/s Ollama-reported decode speed, losing 79.7% of throughput with a median overhead of 8,872 ms per call. The same dataset showed glm-4.7-5090 losing just 2.7% (177 tok/s decode, 172.2 tok/s delivered, 591 ms overhead), which the analysis attributes to VRAM residency and eviction from intermittent calling patterns rather than model architecture. The findings indicate that standard local LLM benchmarks measuring only generation speed miss the reload and queue overhead that determines what applications actually receive.

Every local LLM benchmark you’ve ever read reports one thing: tokens per second during generation. Ollama calls it eval duration . It’s the purest possible measurement – how fast the model spits out tokens once it’s already loaded, already warm, already running. It’s also not what your application receives. We pulled 2,218 instrumented production calls from our own cost logs table over the last 30 days and split every call into two numbers. Decode speed is the Ollama-reported generation rate. Delivered speed is wall-clock – what the calling code actually waited for, queue time and VRAM reload included. The gap between those two numbers is the whole post. What the gap actually looks like Here’s the spread, model by model, decode speed against delivered speed: - phi4:14b : 124.7 tok/s decode, 25.3 tok/s delivered. That’s 79.7% of the advertised throughput gone before it reaches the caller. Median overhead per call: 8,872 ms. 243 calls. - qwen2.5:7b : 236.7 tok/s decode, 105.5 tok/s delivered. 55.4% lost. Overhead 2,068 ms. 114 calls. - qwen3-vl:30b : 162.8 tok/s decode, 84.8 tok/s delivered. 47.9% lost. Overhead 2,636 ms. 940 calls. - gemma-4-31B-it-qat : 62.2 tok/s decode, 36.1 tok/s delivered. 42% lost. Overhead 5,322 ms. 804 calls. - qwen3.6:27b : 124.6 tok/s decode, 99.2 tok/s delivered. 20.4% lost. Overhead 6,019 ms. 83 calls. - glm-4.7-5090:latest : 177 tok/s decode, 172.2 tok/s delivered. Only 2.7% lost. Overhead 591 ms. 34 calls. Look at that last row against the first. phi4:14b decodes slower on paper than qwen2.5:7b and qwen3-vl:30b , but the number that matters – what actually reaches your app – puts it dead last. Meanwhile glm-4.7-5090 sits in the middle of the pack on raw decode speed and comes out on top on delivered speed, losing almost nothing. Same hardware. Same Ollama runtime. A 30-point spread in how much of the advertised speed you actually get to use. It’s not that one model is slower. It’s how often it’s asked to leave The instinct here is to rank these models by “how slow they really are.” That’s the wrong frame, and it’ll send you chasing the wrong fix. The mechanism is residency. A model that’s resident in VRAM answers a request immediately – the GPU already has the weights loaded, decode starts on the first token. A model that gets called intermittently gets evicted between calls, and the next request pays the full cost of loading it back into VRAM before a single token comes out. That reload cost lands entirely inside “overhead,” and overhead is exactly what benchmark decode numbers never measure. glm-4.7-5090 loses almost nothing not because it’s architecturally special, but because our pipeline keeps it warm – it gets called often enough, and pinned deliberately enough, that it rarely gets evicted. phi4:14b loses 80% because it’s the intermittently-invoked model in our stack – the one that sits idle between calls and gets swapped out, so nearly every call pays a full reload. Put phi4:14b on a hot path and that number changes. Put glm-4.7-5090 on a cold path and it’ll post its own ugly overhead figure. The model isn’t the variable. The calling pattern is. This matches something we ran into directly while building speculative decoding for local inference https://www.gladlabs.io/posts/speculative-decoding-for-local-llm-inference-how-a-a5594ce1 – decode-time optimizations only pay off once you’ve already dealt with whatever’s eating time outside the decode loop. A faster draft model doesn’t help you if the target model just got evicted. Why the public benchmarks miss this entirely This isn’t a knock on the benchmark writers – it’s a structural blind spot in how local LLM benchmarks get built. A recent $500 GPU coding benchmark https://www.kunalganglani.com/blog/local-ai-coding-benchmark-ditch-cloud tests Qwen3-Coder against Claude on 50 developer tasks, and a comparison of Qwen3-Coder, Qwen3.6 27B, and Gemma 4 26B https://atomic.chat/blog/guides/best-local-llms-for-coding clocks one model at 220 tokens per second as “the fastest model in our test by a wide margin.” Those numbers are real and useful for what they measure. But they’re measuring a single model, warm, running one task after another, with no eviction pressure and no competing workload fighting for the same GPU. Production doesn’t look like that. In a real pipeline you’ve got several models sharing one card, called at different rates, some hot and some cold, some fighting a queue. A hardware sizing guide for local LLMs will tell you what a mid-range GPU can theoretically push through on Llama 3.3 70B, and a benchmark-ranked model guide will rank models by VRAM tier and coding accuracy. None of that tells you what happens when three of those models are sharing a 5090 and two of them are cold nine times out of ten. What to actually do with this If you’re picking a model off a leaderboard, decode speed tells you the ceiling. It doesn’t tell you what you’ll get. The question that actually matters is: how often will this model be resident when the request comes in? If a model sits on a hot path – called constantly, worth keeping pinned – its decode number and its delivered number will converge, the way glm-4.7-5090 does at 2.7% overhead. If it’s a cold-path model – a critic, a fallback, something invoked occasionally – expect the overhead to dominate regardless of how fast its raw decode looks on paper, the way phi4:14b does at nearly 80%. The fix isn’t picking a different model. It’s deciding, deliberately, what stays resident and what gets evicted, and then measuring the thing you’ll actually experience – wall-clock, queue included – instead of the thing a leaderboard reports. We didn’t get this from reasoning about it. We got it from instrumenting 2,218 real calls and looking at the two columns side by side. If you’re running a multi-model local stack, that’s the only version of this measurement worth trusting.