{"slug": "your-agent-pays-a-tax-on-every-tool-it-never-calls", "title": "Your Agent Pays a Tax on Every Tool It Never Calls", "summary": "Microsoft Foundry introduced Tool Search at Build 2026 to address the token cost and accuracy degradation caused by large tool catalogs in AI agents. The feature uses meta-tools to retrieve a small ranked set of candidates instead of sending the full schema for every tool on each turn, with pinning for frequently used tools to minimize latency. The approach generalizes to any MCP-heavy agent and is most beneficial for catalogs exceeding roughly thirty tools.", "body_md": "Most agents are billed for tools they don't use. Not once — on every single turn.\n\nThe mechanics are simple enough that it's easy to miss. When you give a model a set of tools, the full JSON schema for every tool goes into the request. Names, descriptions, parameter types, enum values, nested objects, the lot. The model reads all of it, picks one, and calls it. Next turn, the whole catalog goes over the wire again, because the API is stateless and the tool list is part of the request.\n\nWith eight tools this is invisible. With two hundred it dominates your input token bill, crowds out the context you actually care about, and — the part that hurts more — measurably degrades tool selection accuracy.\n\nMicrosoft Foundry shipped **Tool Search** at Build 2026 to address exactly this. It's worth understanding, and worth understanding beyond Foundry: the same failure mode shows up in any MCP-heavy agent, and the mitigation generalizes.\n\nA moderately detailed tool schema runs 150–400 tokens once you account for parameter descriptions that are good enough for the model to use correctly. Cheap schemas produce bad tool calls, so teams write generous ones, which is the right call and also the expensive one.\n\nMultiply that across a catalog and across turns:\n\n```\ntokens_per_turn  = base_prompt + conversation_history + (n_tools × avg_schema_tokens)\ntokens_per_task  = tokens_per_turn × turns\n```\n\nA twelve-turn task against a 200-tool catalog at 250 tokens per schema spends roughly 600,000 input tokens on tool definitions alone. The conversation itself might be 20,000. You are paying, overwhelmingly, to re-read a catalog the model already decided against eleven times.\n\nThe token cost is the visible half. The invisible half is worse. As catalogs grow, they accumulate near-duplicates — `get_customer`\n\n, `get_customer_profile`\n\n, `fetch_customer_record`\n\n, `lookup_account_by_customer`\n\n— often from different teams, different MCP servers, different eras of the codebase. Selection accuracy falls not because the model got dumber but because you handed it a genuinely ambiguous menu.\n\nInstead of a flat list, Foundry exposes two meta-tools: `tool_search`\n\nand `call_tool`\n\n. The agent describes what it's trying to do, gets back a small ranked set of candidates, and invokes one.\n\nThe trade is a retrieval round-trip in exchange for not shipping the catalog. Above roughly thirty tools that trade is strongly favourable. Below it, it usually isn't — which is the first thing to be honest about before adopting it.\n\nTool Search isn't the only answer, and it isn't always the right one.\n\n| Strategy | Token cost per turn | Selection accuracy at scale | Added latency | Operational cost |\n|---|---|---|---|---|\n| Flat tool list | Linear in catalog size | Degrades sharply past ~50 tools | None | Trivial |\n| Hand-partitioned catalogs per task type | Low within a partition | Good, if routing is correct | None | High — routing rules rot as tools change |\n| Multi-agent split by domain | Low per sub-agent | Good within domains, poor across them | Handoff overhead | High — orchestration and shared state |\n| Tool Search | Roughly flat regardless of catalog size | Depends on retrieval quality | One extra round trip | Low — index is maintained for you |\n| Tool Search plus pinning | Flat, plus pinned schemas | Best available: hot path guaranteed, tail retrieved | Only on the tail | Low |\n\nPinning is the part people skip and shouldn't. Foundry lets you pin critical tools so they bypass the search round-trip entirely, add context describing how your team actually thinks about a tool, and auto-pin frequently used ones. In practice a handful of tools account for most calls; pin those, retrieve everything else, and you get flat token cost without paying retrieval latency on the common path.\n\nTwo commands scaffold a hosted agent with a toolbox attached.\n\n```\nmkdir my-toolbox-agent && cd my-toolbox-agent\n\nazd ai agent init \\\n  -m \"https://github.com/microsoft-foundry/foundry-samples/blob/main/samples/python/hosted-agents/agent-framework/responses/04-foundry-toolbox/agent.manifest.yaml\" \\\n  --src src/toolbox-agent\n```\n\nThen create the toolbox from the sample's descriptor:\n\n```\nazd ai toolbox create my-toolbox --from-file ./src/toolbox-agent/toolbox.yaml\n```\n\nThat prints a versioned MCP endpoint, which is what your agent binds to:\n\n```\nhttps://<account>.services.ai.azure.com/api/projects/<project>/toolboxes/my-toolbox/versions/1/mcp?api-version=v1\n```\n\nOne gotcha worth flagging because it cost me twenty minutes: `azd ai toolbox create`\n\nneeds a local azd project and environment to run against, even when you pass `--project-endpoint`\n\nexplicitly. If you aren't starting from `azd ai agent init`\n\n, run `azd init --minimal`\n\nfirst and set the endpoint into the environment:\n\n```\nazd init --minimal\nazd env set FOUNDRY_PROJECT_ENDPOINT https://<account>.services.ai.azure.com/api/projects/<project>\n```\n\nYour `toolbox.yaml`\n\nis where the catalog and its search behaviour are declared. The shape below is illustrative — preview schemas move, so check the current sample before copying:\n\n```\n# toolbox.yaml — illustrative structure, verify against the current sample\nname: my-toolbox\ndescription: Order operations, customer lookup, and fulfilment tools\n\ntoolSearch:\n  enabled: true\n  # Pinned tools skip retrieval entirely and are always in context.\n  # Keep this list short — every pin is a permanent token cost.\n  pinned:\n    - get_order_status\n    - search_customers\n  autoPin:\n    enabled: true\n    minCallsPerWindow: 25\n\ntools:\n  - name: get_order_status\n    source: mcp\n    server: orders-mcp\n    # Retrieval context: describe the tool the way your team describes it,\n    # including the vocabulary users actually type.\n    searchContext: >\n      Look up the current fulfilment state of a single order. Use when the\n      user mentions an order number, tracking number, \"where is my package\",\n      or asks whether something shipped.\n\n  - name: issue_refund\n    source: mcp\n    server: billing-mcp\n    searchContext: >\n      Issue a full or partial refund against a completed order. Requires an\n      order ID and an amount. Do not use for cancellations of unshipped\n      orders — use cancel_order instead.\n```\n\nThat last `searchContext`\n\nis doing real work. Retrieval quality is the entire ballgame with Tool Search, and retrieval runs against your descriptions. The explicit negative — *don't use this one, use that one* — is the single highest-value thing you can write there, because near-duplicates are precisely where selection fails.\n\nThe reason this topic is worth writing about rather than just enabling is that the win is measurable, and the size of the win depends entirely on your catalog. Wire up tracing before you change anything and get a baseline.\n\n```\npip install azure-ai-projects azure-identity opentelemetry-sdk azure-core-tracing-opentelemetry\npython\nfrom azure.core.settings import settings\nfrom opentelemetry import trace\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter\nfrom azure.ai.projects.telemetry import AIProjectInstrumentor\n\nsettings.tracing_implementation = \"opentelemetry\"\n\nspan_exporter = ConsoleSpanExporter()\ntracer_provider = TracerProvider()\ntracer_provider.add_span_processor(SimpleSpanProcessor(span_exporter))\ntrace.set_tracer_provider(tracer_provider)\n\n# Emits GenAI spans for every agent and model call in this process\nAIProjectInstrumentor().instrument()\n```\n\nSet `AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING=true`\n\nbefore running, and every turn — prompts, tool calls, model responses — surfaces as a span with token usage attached. Swap `ConsoleSpanExporter`\n\nfor the Azure Monitor exporter and the same spans land in the Foundry portal's Observability tab.\n\nNow run the comparison. The harness below is deliberately dumb: a fixed task set, both configurations, aggregate the spans.\n\n``` python\nimport os\nimport json\nfrom dataclasses import dataclass, field\nfrom statistics import mean\n\n@dataclass\nclass RunResult:\n    config: str\n    input_tokens: list = field(default_factory=list)\n    turns: list = field(default_factory=list)\n    correct_tool: list = field(default_factory=list)\n\nTASKS = [\n    # (prompt, tool the model should end up calling)\n    (\"Where is order 88231?\",                      \"get_order_status\"),\n    (\"Refund the second item on order 88231\",      \"issue_refund\"),\n    (\"Cancel order 90114, it hasn't shipped\",      \"cancel_order\"),\n    (\"Which customers in Ohio ordered twice?\",     \"search_customers\"),\n    (\"Send the June invoice to billing@acme.com\",  \"email_invoice\"),\n]\n\ndef score_run(spans, expected_tool):\n    \"\"\"Pull token usage and the actually-invoked tool out of collected spans.\"\"\"\n    input_tokens = sum(\n        s.attributes.get(\"gen_ai.usage.input_tokens\", 0) for s in spans\n    )\n    invoked = [\n        s.attributes.get(\"gen_ai.tool.name\")\n        for s in spans\n        if s.attributes.get(\"gen_ai.tool.name\")\n    ]\n    # With Tool Search the target arrives as a call_tool argument, so unwrap it.\n    resolved = [\n        json.loads(s.attributes[\"gen_ai.tool.arguments\"]).get(\"name\", n)\n        if n == \"call_tool\" else n\n        for s, n in zip(spans, invoked)\n    ]\n    return input_tokens, len(invoked), expected_tool in resolved\n\ndef report(results):\n    for r in results:\n        print(f\"\\n{r.config}\")\n        print(f\"  mean input tokens/task : {mean(r.input_tokens):>8,.0f}\")\n        print(f\"  mean turns/task        : {mean(r.turns):>8.1f}\")\n        print(f\"  tool selection accuracy: {mean(r.correct_tool):>8.1%}\")\n```\n\nRun it against a flat catalog, then against the same catalog with Tool Search on, then again with your top five tools pinned. Three numbers, one table, and you'll know whether this is worth doing for *your* catalog rather than for a catalog in a blog post.\n\nBeing straight about the limits, because the failure modes are real:\n\n**Small catalogs get worse, not better.** Under about thirty tools, you've added a round-trip and a retrieval failure mode to save tokens you weren't spending. Don't.\n\n**Retrieval misses are silent and confusing.** When a flat-list agent picks the wrong tool, the trace shows it considering the right one. When Tool Search never surfaces the right tool, the trace shows an agent that behaved reasonably given what it was handed. Debugging shifts from \"why did it choose badly\" to \"why wasn't this in the candidate set,\" which is a different skill and a less obvious one.\n\n**Latency moves to the wrong place.** The extra round-trip lands at the start of a turn, before any useful work. For a background agent that's free. For anything a human is watching, pin aggressively.\n\n**Your descriptions are now load-bearing infrastructure.** A vague `searchContext`\n\nused to cost you occasional bad tool calls. Now it costs you tools that are functionally invisible. Budget review time for them the way you'd budget it for an API contract.\n\nStrip Foundry out of this and the principle stands: **anything you send on every turn should earn its place on every turn.** Tool schemas were the first thing to hit this wall because they grow with integration count, but the same audit applies to system prompts that accumulated instructions nobody has read in six months, few-shot examples kept for a model version you no longer run, and retrieved context that's pasted in wholesale because trimming it was somebody's Q3 task.\n\nTool Search is a good implementation of a boring idea — retrieve instead of broadcast. The reason to adopt it isn't that it's new. It's that you can measure the difference in an afternoon, and the number is usually larger than you expect.\n\n*Preview APIs in this area are moving quickly — the azd commands and tracing setup above are current as of the June 2026 Foundry release, but verify schema-level details against the current samples before shipping.*", "url": "https://wpnews.pro/news/your-agent-pays-a-tax-on-every-tool-it-never-calls", "canonical_source": "https://dev.to/jubinsoni/your-agent-pays-a-tax-on-every-tool-it-never-calls-960", "published_at": "2026-08-02 05:35:18+00:00", "updated_at": "2026-08-02 06:10:44.316770+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "ai-infrastructure", "ai-agents", "large-language-models"], "entities": ["Microsoft Foundry", "Build 2026", "Tool Search", "MCP"], "alternates": {"html": "https://wpnews.pro/news/your-agent-pays-a-tax-on-every-tool-it-never-calls", "markdown": "https://wpnews.pro/news/your-agent-pays-a-tax-on-every-tool-it-never-calls.md", "text": "https://wpnews.pro/news/your-agent-pays-a-tax-on-every-tool-it-never-calls.txt", "jsonld": "https://wpnews.pro/news/your-agent-pays-a-tax-on-every-tool-it-never-calls.jsonld"}}