Every connected tool was being described to the model on every call. I tested hiding them behind a search, 96 runs each way. It cut the cost to about a third, and it made the model start inventing ids.
A few days ago someone said "hi" to Doozy in our staging Slack, and I happened to look at what that one word cost. The first model call was 122,598 input tokens. For "hi".
About 5,000 of those were Doozy's instructions. Nearly all the rest was tool definitions. Doozy works in the tools a team connects, and every connected tool was being described to the model, in full, on every call, whether the request needed it or not.
The short version
269 connected tools meant about 118,000 tokens of schema in every model call. Putting them behind a search cut the first call to 8,300 tokens and the whole request to about a third of the cost, with the same share of right answers. It also made the model invent ids 30 times in 96 runs, where it had invented none before. The pass rate didn't show that. The count of refused calls did.
What a "hi" cost #
That workspace has an accounting system, an issue tracker, product analytics, a store and a database connected, plus Slack itself. Together they offer 269 tools, and their definitions come to about 448 KB of JSON.
It isn't spread evenly. Zoho Books alone is 182 of the 269. PostHog is one tool, and that one tool's definition is 42 KB.
And it isn't paid once. A request takes about five model calls, and each one carries the whole catalogue again, so an ordinary request read 636,000 input tokens to write about 1,300. Some model providers cache the repeated part and make it nearly free, which is why I hadn't noticed. Others don't, and the same "hi" costs real money there.
How other agents keep tool schemas out of context #
I went and read how other people deal with this. It comes down to three moves.
- Expose less. Allow-lists per connector, or a hard cap. An accounting connector doesn't need 270 tools in a Slack coworker.
- Defer the schema. The model sees a short list of what's connected and a search tool. A full definition enters the conversation only when the search finds it. Anthropic'stool search works this way.
- Call through code. The tools become a typed API in a sandbox and the model writes code against it, so the definitions never appear as tools at all. Cloudflare'sCode Mode is the clearest write-up.
The second one fit us with the least change, so that's the one I tested. What I couldn't find in any of the write-ups was what it does to the answers. Everyone reports the tokens saved. So I measured the other half.
The test: 96 runs with every tool, 96 with tool search #
I took the real catalogue, all 269 tools exactly as a task is offered them, and put a made-up company behind it: one small business whose books, issues, store, database and analytics agree with each other. The fake systems answer the way the real ones do, errors included. Ask the accounting system for an invoice with an organization id it never gave you and it refuses, as the real one would. That matters, because it's what makes an invented id show up instead of passing quietly.
Then 32 requests of the kind a Slack coworker gets, three times each, in both modes. Find the overdue invoices. Create a coupon with this code and expiry. Look up an order that doesn't exist. Say hi. File an issue in a tracker whose write tools are switched off, where the right answer is to say so.
A run passes when it called the right tool with the right arguments, called nothing it shouldn't, and the reply says what it must and nothing it mustn't. Beside pass or fail I counted the things that look like hallucination whatever the outcome: tool names that don't exist, arguments that don't fit, ids nobody gave it, and calls the system refused.
What held: same answers, a third of the cost #
| 96 runs in each mode, on the model Doozy uses by default | ||
|---|---|---|
| Every tool up front | Tools behind a search | |
| --- | --- | --- |
| Requests answered rightly | 94 of 96 | 94 of 96 |
| Input tokens on the first model call | 118,600 | 8,300 |
| Input tokens for the whole request | 636,000 | 219,000 |
| Model calls per request | 5.3 | 8.4 |
| Seconds per request | 17.7 | 22.3 |
| Ids the model made up | 0 | 30 |
| Calls the systems refused | 5 | 33 |
The top of that table is the good news. The same 94 of 96, the first call down from 118,600 tokens to 8,300, and the whole request at about a third of the cost. It's a little slower, because each search is one more trip to the model: about 22 seconds a request, up from 18.
I also worried it wouldn't find a system nobody named. "How many people visited our website this week?" never says PostHog. With only the word "PostHog" in a list of connected systems to go on, it found the right tool in 12 runs of 12.
What broke: the model started inventing ids #
The bottom of the table is the part I'd have missed if I'd only looked at the pass rate.
Every call to the accounting system needs an organization id, and there's a tool that lists them. With every tool declared, the model could see that tool sitting there, called it first, and used the real id. In 96 runs it never made one up.
Behind a search, it asks for "invoices", gets the invoice tool, and sees a required field called organization id. It hasn't seen the tool that lists them, so it doesn't know there's a step before this one. In 11 runs it filled the field in anyway. 30 invented ids in total.
And they weren't random. Most were the example values written in the tool's own description, the "e.g. 60000000011" kind. The documentation meant to help was where the wrong answer came from.
The model recovered nearly every time. The system refused the guess, it went looking, found the real id and answered correctly, which is why the pass rate didn't move. But those 11 runs took about 60 seconds and 17 model calls each, against 19 seconds and 9 for the rest. And that's the kind case. A customer with several organizations could have a guess that lands on a real one.
One fix worked at once: keep the list-organizations tool always declared, and invented ids went back to zero. I took it out again. It fixes one connector by hand, and the next connector with a setup step would have the same hole. So this is still open, and deferred tools are behind a switch in Doozy for now, not the default.
Three smaller things the runs showed #
Long, repeated tool names get shortened into names that don't exist. In every mode, three or four times in 96 runs, the model called a tool with a made-up suffix on the end. All of them were from the connector whose 182 tool names start with the same 22 characters. The harness refuses a name that isn't offered, so nothing happened, but that's a naming problem on my side, not a model problem.
It wrote junk to find out whether it could write. In the requests where the issue tracker's write tools were off, twice in early runs the model created a label called x and another called probe, to see if it was allowed, before telling the person it couldn't file the issue. In the test that's a row in a fake tracker. In a real one it's a stray label in a customer's workspace. The fix is a written rule against testing access by changing things, because the model won't infer it.
My grader was wrong four times before the model was. It failed a schedule given as structured data instead of text, a refusal that said "couldn't" where I'd matched "can't", and "nothing was created" read as a claim that something was. Each time I only found out by reading the actual replies. If your eval has no transcripts you can read, you're grading your regex.
One more, since I'd rather say it than have you find it. The first runs used the same provider account as production, and they drained its balance far enough that the last runs were refused. No customer task failed before I topped it up, and that was timing, not design. Evals have their own account now.
What I'd do if you have more than a few dozen tools #
- Look at the first model call of an empty conversation. If tool definitions are most of it, that's your bill and probably your latency, on every turn.
- Defer them, and expect the cost to drop by more than half. Keep the handful a task nearly always uses declared. Ours are the Slack tools and web search.
- Find your setup tools before you defer. Any tool whose output every other call needs, an account, an organization, a project, is invisible behind a search. Say so in the description of the tools that depend on it, or keep it loaded.
- Take example ids out of tool descriptions. A model that doesn't have a real value will use the one you showed it.
- Count refused calls, not just right answers. A model that guesses, gets refused and recovers still passes. It also takes three times as long, and one day the guess is valid.
The part I haven't solved is the general one: how a model that finds tools one at a time learns that a tool has a step before it, without me writing that down by hand for every connector. If you've solved it, or you've seen deferred tools go wrong in a different way, write to me at [email protected]. What did your agent make up when it couldn't see the whole list?
Questions people ask #
How many tokens do MCP tool definitions use? #
It depends on the servers, and it adds up fast. Ours came to about 448 KB of JSON schema for 269 tools, which was roughly 113,000 tokens on every model call. One accounting connector was most of that, and a single analytics tool carried 42 KB on its own.
What is deferred tool ? #
The model starts with a short list of what's connected and a search tool, not every tool's full definition. When it needs a capability it searches, and only the tools it finds are loaded into the conversation.
Does tool search make an AI agent less accurate? #
Not on our test. The pass rate was 98% both ways across 96 runs each. What changed was how it failed along the way: with tools found one at a time, the model skipped a setup step and guessed ids it should have looked up.
How many tools is too many for an AI agent? #
Measure the first model call of an empty conversation. If tool definitions are most of it, you have too many declared, whatever the count. For us that point was well behind us at 269.