At Induction AI, we’re building context optimization to make AI agents more efficient. Our service reduces inference costs while preserving task accuracy without requiring changes to the agents themselves.
We run a lot of benchmarks, both to test new optimization ideas and to verify that we’re delivering cost savings without compromising accuracy.
In our early experiments, benchmark trials shared a prompt cache. That made later trials cheaper and skewed our cost measurements.
Why trials share a cache #
When benchmarking agents, we typically run multiple trials of the same task and compute stats on the cost and accuracy distributions.
However, the prompt cache has no idea which conversation a request belongs to. Any request you send where the opening tokens match one the provider processed recently reads that prefix from the cache at a discount (OpenAI, Anthropic).
Trials of the same or related tasks start with the same tools, the same system prompt and the same task message, and often the same first few tool calls. So if trial 2 reads trial 1’s opening turns from the cache, the test would not provide an accurate cost. That includes trials you launch in parallel because the entry exists as soon as the first request is processed.
In production, this is good. But for benchmarks, we want the trials isolated. What we want is caching within a trial, but not across trials. Each trial should start with a cold application-context cache, while successive turns should benefit from normal prompt caching.
Our solution: put the trial id in a tool #
Our cache invalidation simulator shows that tools sit at the very top of the prefix and that adding a tool leaves nothing cached.
So the answer to isolation is to give each trial a tool whose name carries a trial id:
- Different across trials, so no trial matches another’s prefix from its first tokens.
- The same for the whole trial, so the trial’s own turns still cache against each other the way a real conversation does. That is part of the cost you are trying to measure. A fresh id per request would miss the cache on every turn.
Tool names allow letters, digits, underscores and hyphens, up to 64 characters (OpenAI, Anthropic), so keep the id short and plain. A UUID fits with room for a prefix.
One gotcha: the model might call your dummy tool
You might expect that a tool with the description “NEVER call this tool” would never be invoked.
We certainly did. But we’ve observed models calling it anyway. So it’s not enough to declare the dummy tool. You also need to implement a harmless response, such as { success: true }, so the trial can continue if the model calls it. Otherwise, a benchmark might fail for reasons unrelated to the task being evaluated.
An example #
Here is an example implementation with the TypeScript AI SDK:
import { randomUUID } from "node:crypto";
import { generateText, stepCountIs, tool } from "ai";
import { z } from "zod";
function cacheBustTool(trialId: string) {
return {
[`cache_bust_${trialId}`]: tool({
description: "Never call this tool.",
inputSchema: z.object({}),
execute: async () => ({ success: true }),
}),
};
}
// Once per trial, not per request.
const trialId = randomUUID();
const result = await generateText({
model,
system,
messages,
tools: { ...cacheBustTool(trialId), ...benchmarkTools },
stopWhen: stepCountIs(50),
});
If your agent calls generateText once per turn instead of letting it loop, create the id where the trial starts and pass it down to every call.
Why it matters #
We use benchmarks such as tau3 and Toolathlon to measure how much context optimization reduces inference costs while maintaining task success rates.
Prompt caching is an important part of real-world agent economics, so we don’t want to disable it entirely. We just want each trial to measure its own costs rather than benefit from previous trials.
Injecting a unique dummy tool turned out to be a simple solution, and a reminder that the infrastructure surrounding an LLM can influence benchmark results as much as the model itself.