{"slug": "we-cut-our-agent-s-token-bill-by-21-one-task-in-four-got-nothing", "title": "We Cut Our Agent's Token Bill by 21%. One Task in Four Got Nothing.", "summary": "A developer at Benchclaw.io reports that progressive disclosure of tool schemas in an AI agent cut input tokens by about 30% and total cost by about 21%, but a task-level breakdown revealed that the average hid a task that cost 12.3% more on one transport. The analysis of 80 runs with gpt-4o showed that deferral made token usage task-dependent rather than run-to-run noisy, with near-zero variance within tasks and wide variance between tasks.", "body_md": "We turned on progressive disclosure for an agent with 20 tools. Deferring the tool schemas cut input tokens by about 30% and total cost by about 21%. Good result, shipped, write the blog post.\n\nThen I broke the numbers down by task, and the average turned out to be hiding something. Three of our four task types saved 21% to 30%. The fourth saved nothing on one transport and cost **12.3% more** on the other.\n\nThis post is about that fourth task, and about why the number you should care about when you defer tool schemas is not the mean.\n\nAll 80 runs are public. Every command below is one I ran while writing this, with its real output pasted in. You can reproduce the whole thing offline for free.\n\nFour business tasks, each needing exactly one tool. Twenty tool schemas registered, always. Two arms:\n\n`DeferredLoadingToolset`\n\n, so the model has to search for a capability and load it before calling itTwenty runs per cell, sequential, no retries, `gpt-4o`\n\nat temperature 0, `parallel_tool_calls=False`\n\n. Run on 2026-08-06 against `pydantic-ai-slim[openai]==2.24.0`\n\n. Both transports, Chat Completions and the Responses API, because tool search executes server-side on one and through a local fallback on the other.\n\nThe whole thing cost $0.279585 to run.\n\n```\ncell     mean_in    stdev    min    max\nA-chat    1361.5     10.3   1350   1372\nB-chat     945.2    190.8    780   1264\nA-resp    1353.8     10.6   1342   1364\nB-resp    1001.0    263.1    807   1477\n```\n\n`A`\n\nis always-on, `B`\n\nis deferred. Input tokens down 30.6% on Chat Completions and 26.1% on Responses. Cost down 21.2% and 16.8%. Correctness was 80/80 exact matches, so nothing broke.\n\nBut look at the standard deviation column. The always-on arms sit within 22 tokens of each other across every run. The deferred arms swing across a 484 and a 670 token range. Deferral made the input size roughly twenty times more variable.\n\nMy first assumption was run-to-run nondeterminism: the model writes a slightly different search query each time, gets back a slightly different set of schemas, and the prompt size wobbles. That assumption was wrong, and checking it is what produced the actual finding.\n\nHere is the check. Group by task inside each cell instead of pooling the cell:\n\n```\ncurl -sL -o bc025.jsonl https://raw.githubusercontent.com/benchclawio/harness/main/results/bc025-progressive-disclosure-2026-08-06/bc025-scored-raw-2026-08-06.jsonl\n\npython3 -c \"\nimport json, statistics as s, collections\nrows=[json.loads(l) for l in open('bc025.jsonl')]\nfor c in ('A-resp','B-resp'):\n    print('==', c)\n    g=collections.defaultdict(list)\n    for r in rows:\n        if r['cell']==c: g[r['task_id']].append(r['metrics']['tokens_in'])\n    for t,v in sorted(g.items()):\n        print(f'   {t:22s} n={len(v)} mean={s.fmean(v):7.1f} sd={s.stdev(v):6.1f}')\n\"\n```\n\nReal output:\n\n```\n== A-resp\n   currency-conversion    n=5 mean= 1342.0 sd=   0.0\n   defect-threshold       n=5 mean= 1364.0 sd=   0.0\n   inventory-reorder      n=5 mean= 1345.0 sd=   0.0\n   shipment-delay         n=5 mean= 1364.0 sd=   0.0\n== B-resp\n   currency-conversion    n=5 mean=  930.2 sd=   2.7\n   defect-threshold       n=5 mean= 1431.8 sd=  95.6\n   inventory-reorder      n=5 mean=  807.0 sd=   0.0\n   shipment-delay         n=5 mean=  835.0 sd=   0.0\n```\n\nTwo things fall out of this.\n\n**Run-to-run variance is near zero in both arms.** Repeat the same task five times under deferral and you mostly get the identical token count, standard deviation 0.0. The nondeterminism I assumed was there is not there, or is small enough not to matter.\n\n**The spread is between tasks, not between runs.** Under always-on, all four tasks land within 22 tokens of each other, because the 20 schemas dominate and the task text barely moves the total. Under deferral that flattening disappears and the tasks spread from 807 to 1,432 tokens.\n\nThat reframes the whole thing. Deferral does not make your cost noisy. It makes your cost **task-dependent**. Always-on charges you the same amount whatever the user asks. Deferred charges you according to what the model decides to search for and load, which means your bill now tracks your traffic mix.\n\nOnce cost is a function of the task, some tasks can come out behind. Here is the per-task cost, both transports:\n\n``` python\npython3 -c \"\nimport json, statistics as s, collections\nrows=[json.loads(l) for l in open('bc025.jsonl')]\ng=collections.defaultdict(list)\nfor r in rows: g[(r['cell'],r['task_id'])].append(r['metrics']['cost_usd'])\nhdr=f\\\"{'task':22} {'A-resp':>10} {'B-resp':>10} {'delta':>8}   {'A-chat':>10} {'B-chat':>10} {'delta':>8}\\\"\nprint(hdr)\nfor t in ('currency-conversion','defect-threshold','inventory-reorder','shipment-delay'):\n    ar=s.fmean(g[('A-resp',t)]); br=s.fmean(g[('B-resp',t)])\n    ac=s.fmean(g[('A-chat',t)]); bc=s.fmean(g[('B-chat',t)])\n    print(f'{t:22} {ar:10.6f} {br:10.6f} {br/ar*100-100:+7.1f}%   {ac:10.6f} {bc:10.6f} {bc/ac*100-100:+7.1f}%')\n\"\n```\n\nReal output:\n\n```\ntask                       A-resp     B-resp    delta       A-chat     B-chat    delta\ncurrency-conversion      0.003795   0.003001   -20.9%     0.003795   0.002791   -26.5%\ndefect-threshold         0.003944   0.004428   +12.3%     0.003840   0.003810    -0.8%\ninventory-reorder        0.003873   0.002707   -30.1%     0.003828   0.002770   -27.6%\nshipment-delay           0.003970   0.002828   -28.8%     0.003846   0.002691   -30.0%\n```\n\n`defect-threshold`\n\nis the outlier. On the Responses API it cost 12.3% **more** with progressive disclosure on. On Chat Completions it saved 0.8%, which after four decimal places is a rounding error and not a saving.\n\nIts deferred prompt came in at 1,431.8 input tokens against 807.0 for the cheapest task in the same cell. That is 625 extra tokens of loaded schema for a task that, like every other task in the suite, needed exactly one tool.\n\nI want to be careful about the mechanism here, because the bundle does not record it. Every deferred run made exactly 2 tool searches and 3 model requests, `defect-threshold`\n\nincluded, so it is not doing extra round-trips. The most likely explanation is that its search matched more of the 20 capabilities and pulled more schemas back into the prompt than the other tasks did. But the raw JSONL logs token counts and the tool that was finally called, not the schema set the search returned, so I cannot prove that from the published data. Treat it as the obvious inference, not a measurement.\n\nThe cost distribution per run makes the point better than a standard deviation does:\n\n``` python\npython3 -c \"\nimport json\nrows=[json.loads(l) for l in open('bc025.jsonl')]\nfor c in ('B-chat','B-resp'):\n    v=sorted(r['metrics']['cost_usd'] for r in rows if r['cell']==c)\n    print(c, [f'{x:.6f}' for x in v]); print()\n\"\n```\n\nReal output:\n\n```\nB-chat ['0.002470', '0.002587', '0.002717', '0.002717', '0.002717', '0.002717', '0.002740', '0.002800', '0.002800', '0.002800', '0.002815', '0.002845', '0.002845', '0.002845', '0.002845', '0.003810', '0.003810', '0.003810', '0.003810', '0.003810']\n\nB-resp ['0.002707', '0.002707', '0.002707', '0.002707', '0.002707', '0.002828', '0.002828', '0.002828', '0.002828', '0.002828', '0.002992', '0.002992', '0.002992', '0.002992', '0.003037', '0.003922', '0.004497', '0.004573', '0.004573', '0.004573']\n```\n\nThat is not a bell curve with a fat tail. It is a cluster and then a cliff, and the cliff is one task type.\n\nOn the Responses API, 5 of 20 deferred runs cost more than the **average always-on run** ($0.003895). The worst deferred run cost $0.004573, which is 17.4% above the always-on mean. The optimisation that saves 16.8% on average was, for a quarter of these runs, not an optimisation.\n\n**Compute your saving per task type, not per corpus.** A single blended percentage is the one number that cannot tell you whether to ship this. If your traffic is 80% the `defect-threshold`\n\nshape, deferral loses you money while your dashboard reports a saving.\n\n**The saving is capped by the schema share of your prompt.** Deferral removes tool schemas. It does not remove your system prompt, the user message, the conversation history, or the tool results coming back. In our tasks the schemas were roughly a third of the request, so removing nearly all of them saved roughly a third of input tokens. If you have five tools and a 4,000-token system prompt, there is nothing here for you.\n\n**Budget the extra round-trip as a certainty.** Always-on completed in 2 model requests. Deferred took 3, in all 40 deferred runs, on both transports. Not an average with spread, a constant. If your latency budget is per-request rather than per-token, you are buying a variable token reduction with a fixed 50% increase in requests.\n\n**Do not verify deferral through the framework's own view of its tools.** `AgentInfo.function_tools`\n\nlists a deferred tool both before and after it loads, and the local `search_tools`\n\nfallback is present either way. That surface tells you what the agent knows about, not what got serialised to the provider. Every number in this post comes from provider-reported request token counts instead, which is the only thing that maps to the invoice.\n\n**Accuracy was not the thing that broke.** All 80 runs produced exact matches, 20 out of 20 in every cell. I expected correctness to be the risk and it was not, though our tasks each needed exactly one capability. Tasks needing several loads would compound the round-trips and give the model more chances to pick wrong. We did not test that.\n\nThe bundle is 80 raw runs, both manifests with SHA-256s, the deterministic task-suite generator, the worker, the collector and the analysis script:\n\n[https://github.com/benchclawio/harness/tree/main/results/bc025-progressive-disclosure-2026-08-06](https://github.com/benchclawio/harness/tree/main/results/bc025-progressive-disclosure-2026-08-06)\n\n`python3 analyze_bc025.py`\n\nregenerates the confidence intervals. `python3 bc025_capabilities.py`\n\nregenerates the task suite byte for byte. Both are offline and cost nothing, because they read the recorded runs rather than calling a model.\n\nThe full benchmark, including the bootstrap confidence intervals and the part about where the widely quoted \"90% to 98% savings\" figure comes from, is written up at [benchclaw.io](https://benchclaw.io/agent-progressive-disclosure-token-cost/).\n\nOne caveat on the version, since it moved while we were running: `pydantic-ai-slim`\n\n2.25.0 landed on PyPI hours before these runs. We benchmarked 2.24.0 and then diffed the tags. `_tool_search.py`\n\nand `toolsets/deferred_loading.py`\n\n, the entire mechanism under test, are unchanged between them. We have not re-run on 2.25.0.", "url": "https://wpnews.pro/news/we-cut-our-agent-s-token-bill-by-21-one-task-in-four-got-nothing", "canonical_source": "https://dev.to/benchclaw/we-cut-our-agents-token-bill-by-21-one-task-in-four-got-nothing-5h9n", "published_at": "2026-08-18 08:53:02+00:00", "updated_at": "2026-08-18 09:12:47.970028+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-infrastructure", "developer-tools"], "entities": ["Benchclaw.io", "gpt-4o", "pydantic-ai-slim", "Chat Completions", "Responses API"], "alternates": {"html": "https://wpnews.pro/news/we-cut-our-agent-s-token-bill-by-21-one-task-in-four-got-nothing", "markdown": "https://wpnews.pro/news/we-cut-our-agent-s-token-bill-by-21-one-task-in-four-got-nothing.md", "text": "https://wpnews.pro/news/we-cut-our-agent-s-token-bill-by-21-one-task-in-four-got-nothing.txt", "jsonld": "https://wpnews.pro/news/we-cut-our-agent-s-token-bill-by-21-one-task-in-four-got-nothing.jsonld"}}