{"slug": "the-one-time-we-break-the-llm-cache-on-purpose", "title": "The one time we break the LLM cache on purpose", "summary": "Induction AI, a startup building context optimization for AI agents, published a technique for isolating benchmark trials from shared prompt caches by embedding a per-trial UUID in a dummy tool name, such as cache_bust_<trialId>, so each trial starts with a cold cache while its own turns still cache against each other. The company said shared caches made later trials cheaper and skewed cost measurements, since trials of the same task share system prompts, tools and opening tool calls. Induction AI also warned that models sometimes call the dummy tool despite a \"Never call this tool\" description, so it must return a harmless response like { success: true } to keep trials from failing.", "body_md": "At Induction AI, we’re building context optimization to make AI agents more efficient. Our service reduces inference costs while preserving task accuracy without requiring changes to the agents themselves.\n\nWe run a lot of benchmarks, both to test new optimization ideas and to verify that we’re delivering cost savings without compromising accuracy.\n\nIn our early experiments, benchmark trials shared a prompt cache. That made later trials cheaper and skewed our cost measurements.\n\n## Why trials share a cache\n\nWhen benchmarking agents, we typically run multiple trials of the same task and compute stats on the cost and accuracy distributions.\n\nHowever, the prompt cache has no idea which conversation a request belongs to. Any request you send where the opening tokens match one the provider processed recently reads that prefix from the cache at a discount ([OpenAI](https://developers.openai.com/api/docs/guides/prompt-caching), [Anthropic](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)).\n\nTrials of the same or related tasks start with the same tools, the same system prompt and the same task message, and often the same first few tool calls. So if trial 2 reads trial 1’s opening turns from the cache, the test would not provide an accurate cost. That includes trials you launch in parallel because the entry exists as soon as the first request is processed.\n\nIn production, this is good. But for benchmarks, we want the trials isolated. What we want is caching within a trial, but not across trials. Each trial should start with a cold application-context cache, while successive turns should benefit from normal prompt caching.\n\n## Our solution: put the trial id in a tool\n\nOur [cache invalidation simulator](https://www.induction.ai/blog/cache-invalidation) shows that tools sit at the very top of the prefix and that adding a tool leaves nothing cached.\n\nSo the answer to isolation is to give each trial a tool whose name carries a trial id:\n\n- Different across trials, so no trial matches another’s prefix from its first tokens.\n- The same for the whole trial, so the trial’s own turns still cache against each other the way a real conversation does. That is part of the cost you are trying to measure. A fresh id per request would miss the cache on every turn.\n\nTool names allow letters, digits, underscores and hyphens, up to 64 characters ([OpenAI](https://developers.openai.com/api/docs/guides/function-calling), [Anthropic](https://platform.claude.com/docs/en/agents-and-tools/tool-use/implement-tool-use)), so keep the id short and plain. A UUID fits with room for a prefix.\n\n### One gotcha: the model might call your dummy tool\n\nYou might expect that a tool with the description “NEVER call this tool” would never be invoked.\n\nWe certainly did. But we’ve observed models calling it anyway.  So it’s not enough to declare the dummy tool. You also need to implement a harmless response, such as `{ success: true }`, so the trial can continue if the model calls it.  Otherwise, a benchmark might fail for reasons unrelated to the task being evaluated.\n\n## An example\n\nHere is an example implementation with the TypeScript [AI SDK](https://ai-sdk.dev/docs/foundations/tools):\n\n``` js\nimport { randomUUID } from \"node:crypto\";\nimport { generateText, stepCountIs, tool } from \"ai\";\nimport { z } from \"zod\";\n \nfunction cacheBustTool(trialId: string) {\n  return {\n    [`cache_bust_${trialId}`]: tool({\n      description: \"Never call this tool.\",\n      inputSchema: z.object({}),\n      execute: async () => ({ success: true }),\n    }),\n  };\n}\n \n// Once per trial, not per request.\nconst trialId = randomUUID();\n \nconst result = await generateText({\n  model,\n  system,\n  messages,\n  tools: { ...cacheBustTool(trialId), ...benchmarkTools },\n  stopWhen: stepCountIs(50),\n});\n```\n\nIf your agent calls `generateText` once per turn instead of letting it loop, create the id where the trial starts and pass it down to every call.\n\n## Why it matters\n\nWe use benchmarks such as [tau3](https://www.induction.ai/docs/benchmarks/tau3) and [Toolathlon](https://www.induction.ai/docs/benchmarks/toolathlon) to measure how much context optimization reduces inference costs while maintaining task success rates.\n\nPrompt caching is an important part of real-world agent economics, so we don’t want to disable it entirely. We just want each trial to measure its own costs rather than benefit from previous trials.\n\nInjecting a unique dummy tool turned out to be a simple solution, and a reminder that the infrastructure surrounding an LLM can influence benchmark results as much as the model itself.", "url": "https://wpnews.pro/news/the-one-time-we-break-the-llm-cache-on-purpose", "canonical_source": "https://www.induction.ai/blog/benchmark-cache-isolation", "published_at": "2026-10-23 20:41:58+00:00", "updated_at": "2026-10-23 21:12:55.493598+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "ai-infrastructure"], "entities": ["Induction AI", "OpenAI", "Anthropic", "AI SDK", "tau3"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-one-time-we-break-the-llm-cache-on-purpose", "markdown": "https://wpnews.pro/news/the-one-time-we-break-the-llm-cache-on-purpose.md", "text": "https://wpnews.pro/news/the-one-time-we-break-the-llm-cache-on-purpose.txt", "jsonld": "https://wpnews.pro/news/the-one-time-we-break-the-llm-cache-on-purpose.jsonld"}}