{"slug": "do-dynamic-tools-break-prompt-caching", "title": "Do Dynamic Tools Break Prompt Caching?", "summary": "A production audit of 10,186 assistant turns across OpenAI, Google Gemini, and OpenRouter found that dynamic tool activation causes prompt cache misses on only 2.4% to 3.4% of turns, while avoiding 71.2 million unused schema tokens and reducing average turn costs by 32.5%. The study, conducted over 93 multi-turn coding sessions in the Pi coding agent, attributes over 82.5% of cache misses to routine multi-file reads, context compaction, and cloud provider 5-minute idle timeouts, not tool switching.", "body_md": "# Do Dynamic Tools Break Prompt Caching?\n\n**Short answer: No.**\n\nDynamic tool activation does not destroy provider prompt caching. In an audit of 10,186 assistant turns across OpenAI, Google Gemini, and OpenRouter, tool swapping caused a cache miss on **only 2.4% to 3.4% of turns**, while avoiding **71.2 Million unused schema tokens** and reducing average turn costs by **32.5%**.\n\nOver 82.5% of prompt cache misses stem from routine multi-file reads, context compaction, and cloud provider 5-minute idle timeouts, not tool switching.\n\nThis study is the empirical follow-up to my architecture deep-dive:\n\n[How I Cut 80%+ of Context Overhead in My Coding Agent].\n\nWhen I published my architecture for dynamic tool activation in AI coding agents, the most common counterargument on Hacker News was immediate:\n\n\"Adding tools dynamically seems fine, but removing them is a terrible idea. Swapping schemas changes the prompt prefix and constantly nukes your provider prompt cache.\"\n\nOn paper, this objection sounds logical. Major LLM providers (Anthropic, OpenAI, Google) cache prompt prefixes. If you modify the system prompt or tool schemas, the prefix hash changes, and the model must write a fresh cache entry at standard or elevated write pricing.\n\nTo see whether dynamic tool activation actually harms prompt caching in production, I audited **93 multi-turn coding sessions and 10,186 assistant turns** in Pi across OpenAI, Google Gemini, and OpenRouter.\n\nHere is what the empirical telemetry reveals.\n\n## The dataset\n\nI analyzed all interactive coding sessions over an 11-day period where dynamic tool activation was active in daily engineering workflows.\n\n**Total Sessions:** 93 sessions**Total Assistant Turns:** 10,186 turns**Total Tool Calls:** 11,461 executions**Models Tested:** OpenAI (GPT-5.6 Sol, GPT-5.6 Luna, Codex), Google Gemini 3.7 Flash, and OpenRouter community endpoints.**Total Tokens Processed:** 1.22 Billion tokens\n\n## 1. How often do agents actually call standby tools?\n\nThe core assumption behind the \"cache nuke\" concern is that an agent constantly swaps tools in and out every two turns.\n\nIn practice, software development follows a strict power-law distribution.\n\nAcross 11,461 total tool executions:\n\n**Core 4 tools (** 10,805 calls (`bash`\n\n,`read`\n\n,`edit`\n\n,`write`\n\n):**94.28%**).** Standby tools activated on demand:**656 calls (** 5.72%**).\n\n```\nCore tools (bash, read, edit, write):  ██████████████████████████████ 94.28%\nStandby tools (browser, loops, image):  █ 5.72%\n```\n\nOver 94% of an agent's work consists of reading files, editing lines, and running shell commands. Because the 4 default tools never leave the prompt, the tool schema prefix remains 100% identical and cached for over 85% to 90% of the entire session.\n\n## 2. What actually causes cache misses?\n\nAcross all 10,186 assistant turns, there were **2,030 cache miss or zero-cache events** (a 19.93% total miss rate, giving an overall 80.08% cache hit rate).\n\nI categorized every single cache miss by its technical trigger:\n\n*Figure 1: Root causes of cache misses across 10,186 assistant turns. Over 82.5% of misses stem from routine file operations and cloud TTLs, not tool switching.*\n\n| Root Cause | Cache Misses | Share of Misses | Share of ALL 10,186 Turns |\n|---|---|---|---|\nLarge File Reads & Context Compaction | 1,422 | 70.05% | 13.96% |\nTool Activation & TTL Expiration | 354 | 17.44% | 3.47% |\nProvider Idle Timeouts (>5 min) | 150 | 7.39% | 1.47% |\nTurn 0 Session Warmup | 89 | 4.38% | 0.87% |\nModel Switching | 15 | 0.74% | 0.15% |\nTotal | 2,030 | 100% | 19.93% |\n\n### Key takeaways from the miss data:\n\n**Tool activation accounts for only 3.47% of turns.** Across all providers and models, swapping tools or letting a tool expire after its 2-turn TTL caused a cache transition in only 354 turns. The other**96.53% of turns** ran with zero tool-related cache disruption.**On OpenAI models, the tool miss rate was 2.40%.** In 4,164 OpenAI turns, tool activation caused exactly 100 cache transitions.**Over 82.5% of cache misses had nothing to do with tools.** The primary driver of cache churn is dumping 2,000-line files into the prompt, which shifts the context boundary and forces session compaction.\n\n## 3. Why cache hit rates vary by provider\n\nWhen comparing hit rates across providers, the differences come down to how each vendor structures its caching engine:\n\n*Figure 2: Empirical cache hit rates across models and provider endpoints.*\n\n| Provider / Model | Total Turns | Cache Hit Rate | Minimum Token Threshold | Documented Cache TTL |\n|---|---|---|---|---|\nOpenAI (GPT-5.6 / Codex) | 4,164 | 86.5% | 1,024 tokens | 30 minutes |\nGoogle (Gemini 3.7 Flash) | 3,683 | 78.8% | 32,768 tokens (32k) | 1 hour |\nOpenRouter / Free Endpoints | 2,237 | 65.2% | Variable | None / Node cycling |\n\n### The Google Gemini 32k threshold\n\nGoogle Gemini's 78.8% hit rate initially looked lower than OpenAI's 86.5%. Looking into Google Cloud's documentation explains why:\n\n- OpenAI begins caching prompts automatically as soon as the input exceeds\n**1,024 tokens**. - Google Gemini's context caching engine requires prompts to exceed\n**32,768 tokens** before caching activates.\n\nIn short sessions or early turns where context was under 32k tokens, Gemini returned `cacheRead: 0`\n\nby design. Out of 772 Gemini cache misses, **471 misses (61.0%)** occurred solely because the prompt had not yet reached Google's 32k threshold. Once sessions grew past 32k tokens, Gemini's cache hit rate climbed past 90%.\n\n## 4. The financial math: cache writes vs. schema dragging\n\nPrompt cache reads are heavily discounted, but they are not free. OpenAI charges **10% to 50% of base input price** for cached reads ($0.30 to $1.25 / MTok on GPT-5.6 / GPT-4o).\n\nWhen you keep 79 static tools loaded in an environment like Codex, you send roughly **12,000 extra schema tokens on every turn**.\n\nHere is the exact financial balance sheet from my 4,164 OpenAI turns:\n\n| Strategy | Extra Schema Tokens Dragged | Cache Rebuild Cost | Cache-Read Fees | Net Cost Impact |\n|---|---|---|---|---|\nStatic Tools (79 tools loaded 100% of the time) | 49.96 Million tokens | $0.00 | +$62.45 | +$62.45 penalty |\nDynamic Tools (4 core + on-demand standby) | 0 tokens | ~$0.80 (100 writes) | $0.00 | -$61.65 net savings |\n\nDragging 79 static tools forces you to pay a cache-read fee on 12,000 unused tokens on every single turn for the entire day.\n\nBy pruning standby tools after 2 turns of idle time, I spent **~$0.80** on 100 cache rebuilds to save **$62.45** in cache-read fees. That is a **~77x return on investment**.\n\nAcross all models and sessions, my average cost per turn dropped from **$0.0609 to $0.0411 per turn (a 32.5% pure invoice reduction)**.\n\n## How to maximize cache stability in your agent\n\nBased on these 10,000 turns, here are the three rules for maintaining prompt cache stability:\n\n### 1. Promote high-frequency search tools to core\n\nIn my initial setup, `web_search`\n\nand `web_fetch`\n\nwere on standby. Because web search accounted for 36% of all tool activations, promoting both to the default tool set eliminated 36 cache transitions for just ~350 extra tokens.\n\nKeep heavy engines (`browser_use`\n\n, multi-agent loops, image generators) on standby, and leave lightweight, high-frequency lookups in core.\n\n### 2. Enforce deterministic prompt sorting\n\nStandby tool lists should be sorted alphabetically (`standby.sort()`\n\n) before prompt injection. If tool discovery returns items in varying order across runs, the byte string changes and breaks prefix matching.\n\n### 3. Use bounded reads instead of full file dumps\n\nLarge file reads cause 70% of all cache misses. Adding a single guideline to your project instructions stops agents from dumping 2,000-line files:\n\nSearch with\n\n`rg -n`\n\nto locate target lines first, and use`read`\n\nwith`offset`\n\nand`limit`\n\n(100–200 lines) instead of loading entire large files into context.\n\n## Conclusion\n\nDynamic tool activation does not destroy prompt caching.\n\nBecause 94% of coding tasks only use basic file and shell tools, your prompt prefix stays stable across the vast majority of turns. The few cache transitions that do occur cost pennies in cache writes, while saving tens of millions of billed schema tokens and keeping the model's context window clean.", "url": "https://wpnews.pro/news/do-dynamic-tools-break-prompt-caching", "canonical_source": "https://m-reschreiter.at/en/blog/does-dynamic-tool-activation-break-prompt-caching", "published_at": "2026-08-31 10:20:35+00:00", "updated_at": "2026-08-31 10:52:48.432220+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "large-language-models"], "entities": ["OpenAI", "Google Gemini", "OpenRouter", "Pi"], "alternates": {"html": "https://wpnews.pro/news/do-dynamic-tools-break-prompt-caching", "markdown": "https://wpnews.pro/news/do-dynamic-tools-break-prompt-caching.md", "text": "https://wpnews.pro/news/do-dynamic-tools-break-prompt-caching.txt", "jsonld": "https://wpnews.pro/news/do-dynamic-tools-break-prompt-caching.jsonld"}}