{"slug": "the-third-price-i-measured-prompt-caching-across-393-llms-and-found-a-90-hiding", "title": "The Third Price: I Measured Prompt Caching Across 393 LLMs and Found a 90% Discount Hiding Behind One JSON Field", "summary": "A developer measured prompt caching costs across 393 LLMs and found that a single JSON field can unlock a 90% discount on repeated prompts. Tests showed Anthropic's Claude Sonnet required an explicit cache_control marker to cache, while OpenAI's GPT-4o-mini cached automatically, cutting costs by 48% and latency by 30%. The developer warns that without configuration, Anthropic traffic can cost up to 10x more than necessary.", "body_md": "Same model. Same prompt. Ten minutes apart.\n\n```\n$0.00464 per request\n$0.00069 per request\n```\n\nThe difference was one JSON field.\n\nI spent last week measuring what LLM requests actually cost, expecting to write about tokenizer differences. I found something more useful instead.\n\nOpen any LLM pricing page. You get price per input token and price per output token. Every comparison article, every spreadsheet, every \"which model is cheapest\" post uses those two numbers.\n\nProviders publish a third one:\n\n```\nprompt              $0.000002    fresh input\ninput_cache_read    $0.0000002   cached input      <- 10x cheaper\ninput_cache_write   $0.0000025   writing to cache  <- 25% MORE\n```\n\nFor RAG and agent workloads, that third line is where most of your money lives. Here's why. A typical RAG support prompt breaks down like this (measured, not estimated):\n\n```\nsystem prompt          96 tokens   32%   identical on every request\nretrieved context     143 tokens   48%   often stable across a session\nconversation history   55 tokens   18%   prefix is stable\nuser message            6 tokens    2%   the only genuinely new part\n```\n\n**98% of that prompt is repeated content being billed at full price.**\n\nI sent a 2,600-token system prompt to Claude Sonnet twice, ten minutes apart, through an API gateway so both providers ran through identical client code. Then the same test against GPT-4o-mini. Then I pulled pricing for every model on the platform.\n\nThree separate findings came out of it.\n\nFirst run, no cache configuration at all:\n\n```\ncold:  input=2253  cached=0  cost=$0.00470600\nwarm:  input=2251  cached=0  cost=$0.00464200\n```\n\nNothing cached. Full price both times. No error, no warning, no hint that a 90% discount was available.\n\nThen the identical prompt with a `cache_control`\n\nmarker on the system message:\n\n```\nbody = {\n    \"model\": \"anthropic/claude-sonnet-4.5\",\n    \"max_tokens\": 20,\n    \"messages\": [\n        {\n            \"role\": \"system\",\n            \"content\": [{\n                \"type\": \"text\",\n                \"text\": PREFIX,\n                \"cache_control\": {\"type\": \"ephemeral\"}   # <- this line\n            }]\n        },\n        {\"role\": \"user\", \"content\": question},\n    ],\n}\n```\n\nResult:\n\n```\ncold:  input=2628  cached=0     written=2614  cost=$0.00676300\nwarm:  input=2626  cached=2614  written=0     cost=$0.00068680\n```\n\n**90% cheaper on every request after the first.**\n\nNote the first request cost *more* than the uncached version: $0.00676 vs $0.00464, about 46% more. That's the cache write premium. You're ahead by request two, but if you send a prompt exactly once, caching loses you money.\n\nSame test, GPT-4o-mini, no configuration whatsoever:\n\n```\ncold:  input=1355  cached=0      cost=$0.00021525   1737ms\nwarm:  input=1355  cached=1280   cost=$0.00011145   1223ms\n```\n\n1,280 of 1,355 input tokens served from cache. **48% cheaper and 30% faster**, with no code change at all.\n\nSo the discount exists on both providers. On one it's automatic. On the other you have to know to ask, and nothing tells you that you're paying full price.\n\nIf your team assumed caching was automatic everywhere, your Anthropic traffic has been costing up to 10x more than necessary and no signal in the response would have told you.\n\nI changed a single character at the front of the prompt:\n\n```\nwarm (identical prefix):     cached=1280\nbroken (1 char changed):     cached=0\n```\n\nEvery token after that character had to be reprocessed at full price.\n\nThis is documented behavior, but seeing it is different from reading it. Caching matches on the *prefix*, exactly. Anthropic's docs describe it as covering tools, then system, then messages in that order, up to the cache breakpoint. Azure's guidance puts the practical rule bluntly: stable content at the beginning, dynamic content at the end.\n\nWhich means this is a structural property of how you build prompts, not a config flag:\n\n```\n# CACHES WELL\nmessages = [\n    {\"role\": \"system\", \"content\": SYSTEM_PROMPT},     # stable\n    {\"role\": \"system\", \"content\": RETRIEVED_CONTEXT}, # stable per session\n    {\"role\": \"user\", \"content\": user_question},       # changes\n]\n\n# CACHES NOTHING\nmessages = [\n    {\"role\": \"system\", \"content\": f\"Time: {now()}\\n{SYSTEM_PROMPT}\"},  # timestamp\n    {\"role\": \"user\", \"content\": user_question},                        # kills prefix\n]\n```\n\nA timestamp at the top of your system prompt invalidates the cache on every single request. So does a user ID, a session ID, or anything else that varies.\n\nI pulled pricing for all 393 models available on the platform:\n\n```\nmodels priced:        393\nwith cache pricing:   248  (63%)\nwithout:              145\ndiscount   min 1.0x   median 10.0x   max 120.8x\nwrite premium charged by 50 of 71 models publishing one\n\ndistribution:\n  under 5x     68  ##########\n  5-9x         53  ########\n  ~10x        120  ###################\n  11-25x        1\n  over 25x      6\n```\n\nThree things worth pulling out:\n\n**About a third of models don't support caching at all.** That makes cache support a model *selection* criterion, not just a configuration step.\n\n**The median discount is exactly 10x**, but the range runs from 1.0x to 120x. The 1.0x entries — a published cache price identical to fresh input — turned out to be small open-weight models like Granite 8B and `gpt-oss-20b`\n\n, where input is already nearly free. Not traps, just models where caching is economically pointless.\n\n**70% of models publishing a write price charge a premium for it.** This appears to be recent: OpenAI models before the GPT-5.6 family charged nothing for cache writes, and newer ones can.\n\n**Minimum prefix length.** OpenAI skips caching below 1,024 tokens (2,048 on older models). Below the bar you pay full price and get no error — `cached_tokens`\n\njust returns 0. My original test prompt was 317 tokens and would have gotten nothing. I added a guard to my script specifically because I nearly published savings numbers for a prompt that could never have been cached.\n\n**Short TTL.** Cache entries expire after roughly five minutes of inactivity (Anthropic offers a paid one-hour window). There's a subtle trap here: Anthropic measures the lifetime from the *start* of the request, and generation time counts against it. A response that streams for four minutes leaves about one minute for the follow-up to land a hit.\n\n**Routing you don't control.** OpenAI's cached states live on individual machines. Above roughly 15 requests per minute, overflow routing can send your request to a machine with no matching entry. There's a `prompt_cache_key`\n\nparameter to improve that. Your hit rate is partly an infrastructure property, not purely a property of your prompt.\n\nI measured three cost levers on the same workload:\n\n| Lever | Impact | Quality cost |\n|---|---|---|\n| Switch to a cheaper model | up to 26x | changes answers |\nCache a stable prefix |\n39–90% |\nnone |\n| Trim retrieved context 60% | ~10% | changes answers |\n\nCaching is the only one on that list that costs you nothing in quality. Trimming context can change answers. Switching models definitely changes answers. Caching is the same tokens at a lower price.\n\nYet it's the one absent from every \"reduce your LLM costs\" article I've read, all of which are about prompt engineering — the lever measured at about 10%.\n\nHere's the whole verification, about a cent to run:\n\n``` python\nimport os, time, requests\n\nKEY = os.environ[\"OPENROUTER_API_KEY\"]\nUNIT = (\"The customer success team reviews consumption trends weekly to \"\n        \"identify accounts where adoption has plateaued or declined. \")\nPREFIX = UNIT * 45   # ~2,600 tokens, comfortably over the 1,024 minimum\n\ndef go(label, question, use_cache_control=False):\n    system = (\n        [{\"type\": \"text\", \"text\": PREFIX,\n          \"cache_control\": {\"type\": \"ephemeral\"}}]\n        if use_cache_control else PREFIX\n    )\n    body = {\n        \"model\": \"anthropic/claude-sonnet-4.5\",\n        \"max_tokens\": 20,\n        \"usage\": {\"include\": True},\n        \"messages\": [\n            {\"role\": \"system\", \"content\": system},\n            {\"role\": \"user\", \"content\": question},\n        ],\n    }\n    u = requests.post(\n        \"https://openrouter.ai/api/v1/chat/completions\",\n        headers={\"Authorization\": f\"Bearer {KEY}\"},\n        json=body, timeout=120,\n    ).json().get(\"usage\", {})\n    d = u.get(\"prompt_tokens_details\") or {}\n    print(f\"{label}: input={u.get('prompt_tokens')} \"\n          f\"cached={d.get('cached_tokens', 0)} \"\n          f\"written={d.get('cache_write_tokens', 0)} \"\n          f\"cost=${u.get('cost', 0):.8f}\")\n\ngo(\"cold\", \"Summarize in one sentence.\", use_cache_control=True)\ntime.sleep(2)\ngo(\"warm\", \"Summarize in five words.\", use_cache_control=True)\n```\n\nFlip `use_cache_control`\n\nto `False`\n\nand run it again. The gap between those two runs is what you've been leaving on the table.\n\nThe key thing to check is whether `cached_tokens`\n\ncomes back non-zero. Whether you *should* be getting cache hits and whether you *are* getting them are different questions, and only one of them is answerable from a pricing page.\n\nIf you run this and see something different from what I measured, I'd genuinely like to hear about it.\n\nEvery LLM cost comparison uses two prices. There's a third — cached input, typically 10x cheaper — and for RAG and agent workloads that's where most of the money is.\n\nBut only if your prompt clears the minimum, your prefix is byte-identical, you're inside the TTL, and your provider doesn't require you to ask.", "url": "https://wpnews.pro/news/the-third-price-i-measured-prompt-caching-across-393-llms-and-found-a-90-hiding", "canonical_source": "https://dev.to/ptokito/the-third-price-i-measured-prompt-caching-across-393-llms-and-found-a-90-discount-hiding-behind-eck", "published_at": "2026-08-25 16:28:51+00:00", "updated_at": "2026-08-25 17:14:51.364932+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Anthropic", "OpenAI", "Claude Sonnet", "GPT-4o-mini"], "alternates": {"html": "https://wpnews.pro/news/the-third-price-i-measured-prompt-caching-across-393-llms-and-found-a-90-hiding", "markdown": "https://wpnews.pro/news/the-third-price-i-measured-prompt-caching-across-393-llms-and-found-a-90-hiding.md", "text": "https://wpnews.pro/news/the-third-price-i-measured-prompt-caching-across-393-llms-and-found-a-90-hiding.txt", "jsonld": "https://wpnews.pro/news/the-third-price-i-measured-prompt-caching-across-393-llms-and-found-a-90-hiding.jsonld"}}