{"slug": "claude-prompt-caching-why-agent-loops-miss-the-20-block-lookback", "title": "Claude Prompt Caching: Why Agent Loops Miss the 20-Block Lookback", "summary": "An engineer discovered that Claude prompt caching misses its 20-block lookback window in agent loops, causing silent cache misses and 12x cost swings. They detail how parallel tool calls append 18 blocks per turn, exceeding the limit, and propose rotating cache breakpoints at a stride shorter than 20 blocks to maintain cache hits.", "body_md": "Your agent starts a run with `cache_read_input_tokens`\n\nat 40K and climbing. Twelve tool calls later, reads drop to zero and `cache_creation_input_tokens`\n\njumps to the full conversation length — on every single turn. Nothing in your prompt changed. No timestamp, no reordered tool, no model switch. The prefix is byte-identical.\n\nYou just hit the 20-block lookback window, and it is the single most expensive thing about Claude prompt caching that nobody puts in their retro.\n\n`cache_control`\n\nbreakpoint searches backward through `tool_choice`\n\n, images, and toggling thinking preserve the tools+system cache. Only tool-definition changes and model switches force a full rebuild.`{\"role\": \"system\", ...}`\n\nmessage to `messages[]`\n\ninstead (Claude Opus 5, Opus 4.8, Fable 5; not Sonnet 5).`input_tokens`\n\nin the usage block is the `input_tokens + cache_creation + cache_read`\n\n. Dashboards that graph `input_tokens`\n\nalone will show you a flat line while you burn cache writes.Because cache lookup is bounded. Prompt caching is a prefix match on exact bytes, but a breakpoint doesn't scan the entire history for a matching entry — it walks backward a limited number of content blocks. That limit is 20. If the previous request's cached block is more than 20 blocks behind your new breakpoint, the lookup fails, and the API treats your request as cold even though a perfectly valid entry exists.\n\nChat apps never see this. One user turn is one text block; one assistant turn is one text block. You'd need ten round trips to move 20 blocks, and you place a breakpoint on each turn anyway.\n\nAgent loops are different. Count what a single \"turn\" actually appends:\n\n`tool_use`\n\nblocks`tool_result`\n\nblocksAn agent that fires 8 parallel tool calls appends 8 `tool_use`\n\n+ 8 `tool_result`\n\n+ 2 = **18 blocks in one round trip**. Two of those turns and your single trailing breakpoint is 36 blocks past the last cached point. Silent miss. No error, no warning field — just `cache_read_input_tokens: 0`\n\nand a cache-creation charge for the full history.\n\nThe economics are brutal at scale. On Claude Opus 5 at $5/MTok input, a cache read is ~$0.50/MTok and a 5-minute cache write is ~$6.25/MTok. A 12x price swing per turn, triggered by a config detail you never set.\n\nStop putting one marker on the last block. Rotate a small set of markers through the message list at a stride shorter than the lookback window. Every breakpoint is both a write point *and* a read point, so a trailing chain of them means each new request always finds a prior entry within 20 blocks.\n\nThe budget matters: **4 breakpoints per request, total, across tools + system + messages.** Spend one on the last system block (it caches tools and system together, since render order is `tools`\n\n→ `system`\n\n→ `messages`\n\n) and rotate the remaining three.\n\n```\nCACHEABLE = {\"text\", \"image\", \"tool_use\", \"tool_result\", \"document\"}\nSTRIDE = 15          # < 20-block lookback, with headroom\nMSG_BREAKPOINTS = 3  # 4 total minus the one on the system block\n\ndef place_breakpoints(messages):\n    \"\"\"Re-place cache_control so no two markers are >STRIDE blocks apart.\n    Mutates plain-dict messages in place (round-trip SDK objects with\n    .model_dump() first — you cannot set cache_control on a response object).\"\"\"\n    flat = []\n    for m in messages:\n        if isinstance(m[\"content\"], str):\n            m[\"content\"] = [{\"type\": \"text\", \"text\": m[\"content\"]}]\n        for b in m[\"content\"]:\n            b.pop(\"cache_control\", None)   # clear last request's markers\n            flat.append(b)\n\n    marks, pos = [], len(flat) - 1\n    while pos >= 0 and len(marks) < MSG_BREAKPOINTS:\n        # thinking blocks can't carry cache_control — skip back to an eligible one\n        while pos >= 0 and flat[pos].get(\"type\") not in CACHEABLE:\n            pos -= 1\n        if pos < 0:\n            break\n        marks.append(pos)\n        pos -= STRIDE\n\n    for i in marks:\n        flat[i][\"cache_control\"] = {\"type\": \"ephemeral\"}\n    return messages\n\nresp = client.messages.create(\n    model=\"claude-opus-5\",\n    max_tokens=8192,\n    system=[{\"type\": \"text\", \"text\": SYSTEM_PROMPT,\n             \"cache_control\": {\"type\": \"ephemeral\"}}],   # caches tools + system\n    tools=TOOLS,                                          # sorted, frozen for the run\n    messages=place_breakpoints(history),\n    thinking={\"type\": \"adaptive\"},\n)\n```\n\nTwo details that bite:\n\n**Clear the old markers.** Breakpoints you set on turn 5 are still sitting in the history you resend on turn 6. If you don't strip them you'll exceed 4 and get a validation error, or worse, waste your budget on positions that no longer help.\n\n**Thinking blocks are not cacheable anchors.** `cache_control`\n\ngoes on `text`\n\n, `image`\n\n, `tool_use`\n\n, `tool_result`\n\n, and `document`\n\nblocks. Landing a stride on a thinking block and silently dropping the marker is exactly how you end up 20+ blocks apart again.\n\nNot all of them, and this is where most teams over-engineer. There are three tiers — tools, system, messages — and a change only invalidates its own tier and everything after it.\n\n| Change | Tools | System | Messages |\n|---|---|---|---|\n| Tool definitions (add/remove/reorder) | ❌ | ❌ | ❌ |\n| Model switch | ❌ | ❌ | ❌ |\n`speed` , web-search/citations toggle |\n✅ | ❌ | ❌ |\n| System prompt content | ✅ | ❌ | ❌ |\n`tool_choice` , images, thinking on/off |\n✅ | ✅ | ❌ |\n| Message content | ✅ | ✅ | ❌ |\n\nRead the practical consequence: you can force `tool_choice`\n\non one turn, flip thinking off on the next, and keep the tools+system cache intact. Stop threading a \"cache-safe mode\" flag through your call sites for those.\n\nWhat you cannot do casually is touch the tool array. Tools render at position 0, so adding one tool for one turn rebuilds *everything*. Serialize tool definitions deterministically — sort by name, `sort_keys=True`\n\non any JSON schema you generate — because a dict-ordering flip in your schema builder is a full-prefix invalidation that looks like nothing in a diff.\n\nModel switching has no escape hatch; caches are model-scoped. If you want a cheap Haiku 4.5 pass over a long context, that's a separate cache lineage, not a discount on the existing one.\n\nAppend it to `messages[]`\n\nas a system-role message instead of editing the top-level `system`\n\nfield:\n\n```\nhistory.append({\"role\": \"user\", \"content\": \"...\"})\nhistory.append({\n    \"role\": \"system\",\n    \"content\": \"Terse mode enabled — keep responses under 40 words.\",\n})\n```\n\nEditing top-level `system`\n\nchanges bytes ahead of the entire conversation, so every cached turn gets reprocessed at full price. A `role: \"system\"`\n\nmessage sits *after* the history and leaves the cached prefix intact.\n\nIt's supported on Claude Opus 5, Opus 4.8, Fable 5, and Mythos 5 with no beta header — **not** Sonnet 5, which returns a 400. Wrap it and fall back to putting the instruction in a user-turn block.\n\nThere's a security bonus that's easy to miss: this is a non-spoofable operator channel. Instructions embedded as text inside user or tool content can be forged by anything that writes into that content — a scraped page, a tool response, a file. A `role: \"system\"`\n\nmessage cannot.\n\nBecause a cache entry only becomes readable once the first response **begins streaming**. Fire ten identical-prefix requests simultaneously and none of them can read what the other nine are still writing. You pay ten cache writes at 1.25x — worse than not caching at all.\n\nThe fix is a two-phase fan-out: send one request, await the first streamed token (not the full response), then release the remaining N−1. They read the entry the first one just wrote. On a 100K-token shared prefix across ten workers, that's the difference between ten writes and one write plus nine reads.\n\nSame reasoning applies to sub-agents and compaction passes. If a fork rebuilds `system`\n\nor `tools`\n\nwith any difference from the parent, it misses the parent's cache entirely. Copy `model`\n\n, `system`\n\n, and `tools`\n\nverbatim, then append fork-specific content at the end.\n\nProbably the minimum cacheable prefix — and it is **not monotonic across model generations**:\n\n| Model | Minimum |\n|---|---|\n| Claude Opus 5, Fable 5, Mythos 5 | 512 tokens |\n| Opus 4.8, Sonnet 5, Sonnet 4.6, Sonnet 4.5 | 1024 tokens |\n| Opus 4.7, Haiku 3.5 | 2048 tokens |\n| Opus 4.6, Opus 4.5, Haiku 4.5 | 4096 tokens |\n\nBelow the threshold, the marker is accepted and ignored: no error, `cache_creation_input_tokens: 0`\n\n. A 3K-token prefix caches on Opus 5 and Sonnet 4.5, and silently doesn't on Opus 4.6 or Haiku 4.5. Teams that route cheap sub-tasks to Haiku and expensive ones to Opus routinely find their Haiku path has never cached once.\n\nThe other silent killer is TTL. The default entry lives 5 minutes. If a tool call in your loop takes 6 minutes — a long build, a slow scrape, a human approval gate — the entry expires *mid-turn* and the next request rebuilds from scratch. Use `{\"type\": \"ephemeral\", \"ttl\": \"1h\"}`\n\non those paths. The write costs 2x instead of 1.25x, so break-even moves from two requests to three, which is trivially cleared by any agent loop.\n\nLog all three usage fields per request, not one:\n\n```\nu = resp.usage\ntotal_prompt = u.input_tokens + u.cache_creation_input_tokens + u.cache_read_input_tokens\nprint(f\"read={u.cache_read_input_tokens} write={u.cache_creation_input_tokens} \"\n      f\"raw={u.input_tokens} total={total_prompt}\")\n```\n\nThe diagnostic pattern for the lookback bug is unmistakable: `cache_read`\n\ngrows normally for the first few turns, then collapses to 0 while `cache_creation`\n\nequals the full history — and it flips exactly on the turn where a burst of parallel tool calls landed. Correlate the miss with the number of blocks that turn appended and you'll see the 20-block cliff in your own logs.\n\nIf you want the API to tell you directly, there's a cache diagnostics beta: `client.beta.messages.*`\n\nwith the `cache-diagnosis-2026-04-07`\n\nbeta flag, passing `diagnostics: {previous_message_id: <prior response id>}`\n\n. It reports why a request missed instead of leaving you to diff prompt bytes by hand.\n\nClaude prompt caching misses inside agent loops because each `cache_control`\n\nbreakpoint only searches backward 20 content blocks for a matching entry, and a single agentic turn with parallel tool calls easily appends more than 20 blocks — so a lone trailing breakpoint lands out of range and the API rewrites the entire prefix at 1.25x instead of reading it at 0.1x. Fix it by rotating breakpoints through the message list at a ~15-block stride within the 4-marker budget, keeping one marker on the last system block, stripping stale markers before each request, skipping thinking blocks as anchors, and switching to a 1-hour TTL wherever tool latency can exceed five minutes. Then confirm it with `cache_read_input_tokens`\n\nrather than the `input_tokens`\n\nfield, which reports only the uncached remainder and will happily look healthy while your cache does nothing.", "url": "https://wpnews.pro/news/claude-prompt-caching-why-agent-loops-miss-the-20-block-lookback", "canonical_source": "https://dev.to/ji_ai/claude-prompt-caching-why-agent-loops-miss-the-20-block-lookback-d36", "published_at": "2026-08-21 15:25:29+00:00", "updated_at": "2026-08-21 15:45:36.810167+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-agents", "developer-tools"], "entities": ["Claude", "Anthropic", "Claude Opus 5", "Opus 4.8", "Fable 5", "Sonnet 5"], "alternates": {"html": "https://wpnews.pro/news/claude-prompt-caching-why-agent-loops-miss-the-20-block-lookback", "markdown": "https://wpnews.pro/news/claude-prompt-caching-why-agent-loops-miss-the-20-block-lookback.md", "text": "https://wpnews.pro/news/claude-prompt-caching-why-agent-loops-miss-the-20-block-lookback.txt", "jsonld": "https://wpnews.pro/news/claude-prompt-caching-why-agent-loops-miss-the-20-block-lookback.jsonld"}}