{"slug": "your-cache-hit-rate-is-low-true-and-worth-0-16", "title": "\"Your cache hit rate is low\" — true, and worth $0.16", "summary": "Anthropic's prompt caching cut API input costs by 83.9% for Traceguard's Claude Code sessions, saving $62,934 over 71 days, according to an analysis by the Traceguard team. The team found that while a low cache hit rate on direct API calls was structurally unavoidable, the excluded Claude Code traffic benefited massively from caching, and a proposed cache-warming ping strategy would lose money on their data.", "body_md": "Anthropic emailed our org last week: prompt cache hit rate is low, and caching repeated content \"could save up to 64% of direct API spend.\" The email is careful with scope — direct API only, Claude Code excluded because it manages caching itself. I went to check what that 64% was made of.\n\nOur direct API traffic over the period: 16 calls, $0.25 total, average prompt 90 tokens. They come from traceguard's rerun harness, which replays self-contained consults as fresh single-turn calls — every prompt unique by construction. A workload like that cannot hit cache: there is no repeated prefix across calls, and 90 tokens is far below the minimum Anthropic will cache at all (1,024 tokens on Opus 4.8, the model those calls ran on; 512 on Opus 5 / Fable 5). Hit rate 0%, structurally. 64% of $0.25 is $0.16.\n\nThe metric was right. The money wasn't there.\n\nThe interesting numbers sit in the traffic the email correctly excluded. We ingest our own Claude Code session logs into traceguard's trace store (Claude Code transcripts carry no cost field, so we compute list price from `usage`\n\n). 71 days, 158 sessions, 58,753 API messages. This is section 1 of the command output, unedited:\n\n| model | messages | prompt tok | hit rate | input cost | no-cache | saved | saved % |\n|---|---|---|---|---|---|---|---|\n| claude-opus-4-8 | 23,759 | 5,535,807,891 | 96.1% | $4,616.39 | $27,691.44 | $23,075.05 | 83.3% |\n| claude-fable-5 | 15,291 | 3,583,420,473 | 96.2% | $5,847.08 | $35,834.20 | $29,987.13 | 83.7% |\n| claude-opus-5 | 6,316 | 1,750,760,343 | 97.7% | $1,205.08 | $8,753.80 | $7,548.72 | 86.2% |\n| claude-sonnet-5 | 11,139 | 1,326,223,701 | 95.1% | $422.96 | $2,652.45 | $2,229.49 | 84.1% |\n| claude-haiku-4-5-20251001 | 1,573 | 57,312,161 | 93.9% | $9.76 | $57.31 | $47.56 | 83.0% |\n| claude-opus-4-7 | 64 | 11,045,966 | 96.6% | $9.04 | $55.23 | $46.19 | 83.6% |\n| claude-sonnet-4-5-20250929 | 2 | 42,410 | 0.0% | n/a | n/a | n/a | n/a |\n| (none) | 609 | 0 | n/a | n/a | n/a | n/a | n/a |\n| TOTAL | 58,753 | 12,264,612,945 | 96.2% | $12,110.31 | $75,044.44 | $62,934.13 | 83.9% |\n\nHit rate is token-weighted; costs are the input side only, at list price. The two n/a rows are deliberate — no published price for a model means no guessed money, tokens counted anyway.\n\nCaching nobody at our org configured cuts the input side by 83.9%. With output included, the whole bill lands at 1/5.5 of the no-cache counterfactual. There is nothing left to optimize here; the remaining spend is mostly first-write premiums on genuinely new content.\n\nThere's a folk optimization going around: ping your session on a timer so the cache stays warm and the next turn hits. The mechanics say it could work — the cache TTL is 5 minutes by default, every hit refreshes it for free, and a 1-hour TTL costs 2× on writes. Instead of arguing, we ran the counterfactual on our own gap data.\n\n97.3% of our in-session gaps are under 5 minutes; they need nothing. The addressable prize is 422 gaps longer than an hour: the context rewrites right after them cost at most $1,913 over 71 days (an overestimate, since it counts genuinely new content as rewrite). Bridging those same gaps with a ping every 55 minutes would have taken 6,765 pings costing $2,009 in cache reads alone, before counting the pings' own output tokens. The strategy loses money on our data, under assumptions tilted in its favor.\n\nThe reason is boring: the 1-hour TTL removed the ping's niche. Our logs show Claude Code already routes nearly all cache writes into the 1h bucket (opus-4-8: 155.9M tokens written at 1h TTL vs 56.4M at 5m). Pinging is a hack from before extended TTL existed. If a tool sells you auto-ping in 2026, ask it for the counterfactual on your logs first.\n\nWhile writing this I checked our own SDK and found the same class of bug I keep filing against other trackers. traceguard's `wrap_anthropic`\n\nrecorded `tokens_in = usage.input_tokens`\n\n. On Anthropic's API, `input_tokens`\n\nexcludes cache reads and writes — and on our traffic, cache reads run about 1,000× larger than raw input, so the wrapper under-counted by three orders of magnitude. Every field read correctly, the quantity still wrong (the splitrail #220 pattern). Fixed in [PR #39](https://github.com/lizhuojunx86/traceguard/pull/39), and the fix is what makes the next part possible.\n\nA rate without a denominator is not a finding. Percentages travel well in emails; dollars make decisions. Before acting on a metric email, multiply it by the money it applies to.\n\nThis analysis is now one command, merged in [PR #40](https://github.com/lizhuojunx86/traceguard/pull/40): `python -m traceguard.routing_audit.cache_audit`\n\n. Point it at your ingested Claude Code transcripts or a traceguard trace store, and get the table above for your own data — per-model hit rate, actual vs no-cache cost at list price, gap distribution, and a yes/no on whether keep-alive pings would pay for you (ours printed NOT WORTH IT). Ships in [traceguard 1.2.0](https://pypi.org/project/traceguard/1.2.0/) — `pip install -U traceguard`\n\n. The last line of ours came out like this:\n\n```\nClaude Code caching already saves us 84% ($12,110.31 vs $75,044.44 list). Checked with: python -m traceguard.routing_audit.cache_audit\n```\n\nIf you run it, send me your numbers. Especially if pings win somewhere — I'd like to see the traffic shape that makes them rational.\n\nLi Zhuojun", "url": "https://wpnews.pro/news/your-cache-hit-rate-is-low-true-and-worth-0-16", "canonical_source": "https://dev.to/lizhuojunx86/your-cache-hit-rate-is-low-true-and-worth-016-30ie", "published_at": "2026-08-16 03:45:04+00:00", "updated_at": "2026-08-16 04:41:23.304076+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools", "mlops"], "entities": ["Anthropic", "Claude Code", "Traceguard", "Claude Opus 4.8", "Claude Opus 5", "Claude Fable 5", "Claude Sonnet 5", "Claude Haiku 4.5"], "alternates": {"html": "https://wpnews.pro/news/your-cache-hit-rate-is-low-true-and-worth-0-16", "markdown": "https://wpnews.pro/news/your-cache-hit-rate-is-low-true-and-worth-0-16.md", "text": "https://wpnews.pro/news/your-cache-hit-rate-is-low-true-and-worth-0-16.txt", "jsonld": "https://wpnews.pro/news/your-cache-hit-rate-is-low-true-and-worth-0-16.jsonld"}}