{"slug": "why-output-caching-doesn-t-work", "title": "Why output caching doesn't work", "summary": "Caching LLM outputs is not worth the effort because only about 6% of output tokens are visible to users, according to an analysis by software engineer Sean Goedecke. Goedecke estimates that 50% of output tokens are reasoning tokens and 44% are tool calls, so even saving two-thirds of visible output would cut total output tokens by just 4% under Amdahl's law. The real savings lie in reducing the 94% of invisible output tokens, which depends on model intelligence: a model 5x more expensive per token can be cheaper overall if it reaches the right answer with 6x fewer tokens.", "body_md": "LLM inference economics depends on caching<sup>[1](https://seangoedecke.com/why-output-caching-doesnt-work/#fn-1)</sup> model inputs, to prevent the entire conversation history being recomputed on each message<sup>[2](https://seangoedecke.com/why-output-caching-doesnt-work/#fn-2)</sup>. Should we cache model *outputs* in the same way? As it turns out, no.\n\nFirst, here’s why we might think caching outputs is a good idea. We cache inputs to prevent repeated work, but LLMs do a lot of repeated work in their output as well:\n\nIn all of these cases<sup>[3](https://seangoedecke.com/why-output-caching-doesnt-work/#fn-3)</sup> I could go and read the files myself, but I don’t. Agents produce such a volume of output, and reading agent output is tiresome, so it’s better to simply ask for the key points.\n\nIt’s an obvious waste of tokens<sup>[4](https://seangoedecke.com/why-output-caching-doesnt-work/#fn-4)</sup> to repeat all of this content. In fact, it’s worse than the input caching situation, because it’s a waste of *output* tokens, which are significantly more expensive to produce<sup>[5](https://seangoedecke.com/why-output-caching-doesnt-work/#fn-5)</sup> than input tokens are to ingest.\n\nIf you want to prevent this, you’d have to get the model to avoid repeating itself. For instance, you could fine-tune or prompt a coding model to output “src/script.py:12-15” instead of simply regurgitating those lines of the file.\n\nMore generally, you could fine-tune a model to reference segments of its own conversation history instead of repeating them. The harness could then trivially expand those references with the actual content, so all of this is transparent to the user.\n\nUnfortunately none of this is a good idea, because **it only optimizes visible output**. I don’t *think* models repeat themselves often in their internal chain of thought<sup>[6](https://seangoedecke.com/why-output-caching-doesnt-work/#fn-6)</sup>. I can’t read frontier chains of thought, but my sense from open models and general principles is that they shouldn’t need to, because that information is already available to them. So optimizing repetition only optimizes the segment of model output that is visible to the user. How much is that?\n\nBased on my own agent histories, it’s about 6%. 50% of output tokens are reasoning tokens, and the remaining 44% are tool calls<sup>[7](https://seangoedecke.com/why-output-caching-doesnt-work/#fn-7)</sup>. So we’re running headlong into [Amdahl’s law](https://en.wikipedia.org/wiki/Amdahl%27s_law): even if you can save two-thirds of the tokens in visible output (a ridiculously optimistic estimate), you’re still only saving 4% of total output tokens.\n\nIf you want to save output tokens, you need to reduce the 94% of output tokens that are invisible, which largely depends on *intelligence*. The smarter the model<sup>[8](https://seangoedecke.com/why-output-caching-doesnt-work/#fn-8)</sup>, the fewer reasoning tokens and tool calls it needs to get to the right answer. This is why we see a strange phenomenon<sup>[9](https://seangoedecke.com/why-output-caching-doesnt-work/#fn-9)</sup> where more expensive models are actually cheaper to use: they might be 5x more expensive per-token, but they can get to the right answer with 6x fewer tokens, so the overall cost is less.\n\nIn the KV cache.\n\nIncluding tool calls.\n\nCoding agents often used to repeat tokens in tool calls (e.g. by re-outputting the entire file content for each edit) but I think this behavior has mostly been replaced with more efficient find-and-replace editing tools.\n\nI’m pretty sure that most mature inference providers already do speculative decoding based on previous ngrams in the input, which would at least exploit output repetition to deliver a speedup.\n\nThe reason, broadly speaking, is also the reason why transformers are so important: because of the attention mechanism, input tokens can be ingested in parallel, while output tokens must be generated one-at-a-time.\n\nI’m only moderately confident about this. Some open-source reasoning models (particularly early ones) do, but those models also use way more tokens than frontier models for the same output, so my guess is that powerful frontier models don’t loop in their chain of thought very often.\n\nIt’s possible that tool calls involve some repetition as well, if agents are composing content in the chain of thought and then repeating it in the tool. Any readers who have done a lot of coding with open-source models (or who work at a frontier lab) and thus know the answer to this, please email me if you can — I’m curious.\n\nAnd the better the harness, though I remain convinced that there isn’t that much comparative advantage in harness quality because everyone can copy each other.\n\nThis applies to real life too, and is the reason why you shouldn’t hire the cheapest plumber you can find.", "url": "https://wpnews.pro/news/why-output-caching-doesn-t-work", "canonical_source": "https://seangoedecke.com/why-output-caching-doesnt-work/", "published_at": "2026-11-20 23:13:32.711745+00:00", "updated_at": "2026-11-20 23:13:35.286137+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-infrastructure"], "entities": ["Sean Goedecke", "Amdahl's law"], "also_reported_by": [], "takeaway": "Visible output is only about 6% of agent output tokens (50% reasoning, 44% tool calls), so caching or de-duplicating it saves at most 4% of total output tokens; cutting cost means reducing reasoning and tool-call tokens via smarter models.", "recommendation": {"advice": "Don't build output caching or de-duplication to cut agent token spend; target the 94% of invisible reasoning and tool-call tokens instead, since a model 5x pricier per token can be cheaper at 6x fewer tokens.", "playbook": "llm-cost", "playbook_url": "https://wpnews.pro/playbooks/llm-cost"}, "alternates": {"html": "https://wpnews.pro/news/why-output-caching-doesn-t-work", "markdown": "https://wpnews.pro/news/why-output-caching-doesn-t-work.md", "text": "https://wpnews.pro/news/why-output-caching-doesn-t-work.txt", "jsonld": "https://wpnews.pro/news/why-output-caching-doesn-t-work.jsonld"}}