Most agents are billed for tools they don't use. Not once — on every single turn.
The mechanics are simple enough that it's easy to miss. When you give a model a set of tools, the full JSON schema for every tool goes into the request. Names, descriptions, parameter types, enum values, nested objects, the lot. The model reads all of it, picks one, and calls it. Next turn, the whole catalog goes over the wire again, because the API is stateless and the tool list is part of the request.
With eight tools this is invisible. With two hundred it dominates your input token bill, crowds out the context you actually care about, and — the part that hurts more — measurably degrades tool selection accuracy.
Microsoft Foundry shipped Tool Search at Build 2026 to address exactly this. It's worth understanding, and worth understanding beyond Foundry: the same failure mode shows up in any MCP-heavy agent, and the mitigation generalizes.
A moderately detailed tool schema runs 150–400 tokens once you account for parameter descriptions that are good enough for the model to use correctly. Cheap schemas produce bad tool calls, so teams write generous ones, which is the right call and also the expensive one.
Multiply that across a catalog and across turns:
tokens_per_turn = base_prompt + conversation_history + (n_tools × avg_schema_tokens)
tokens_per_task = tokens_per_turn × turns
A twelve-turn task against a 200-tool catalog at 250 tokens per schema spends roughly 600,000 input tokens on tool definitions alone. The conversation itself might be 20,000. You are paying, overwhelmingly, to re-read a catalog the model already decided against eleven times.
The token cost is the visible half. The invisible half is worse. As catalogs grow, they accumulate near-duplicates — get_customer
, get_customer_profile
, fetch_customer_record
, lookup_account_by_customer
— often from different teams, different MCP servers, different eras of the codebase. Selection accuracy falls not because the model got dumber but because you handed it a genuinely ambiguous menu.
Instead of a flat list, Foundry exposes two meta-tools: tool_search
and call_tool
. The agent describes what it's trying to do, gets back a small ranked set of candidates, and invokes one.
The trade is a retrieval round-trip in exchange for not shipping the catalog. Above roughly thirty tools that trade is strongly favourable. Below it, it usually isn't — which is the first thing to be honest about before adopting it.
Tool Search isn't the only answer, and it isn't always the right one.
| Strategy | Token cost per turn | Selection accuracy at scale | Added latency | Operational cost |
|---|---|---|---|---|
| Flat tool list | Linear in catalog size | Degrades sharply past ~50 tools | None | Trivial |
| Hand-partitioned catalogs per task type | Low within a partition | Good, if routing is correct | None | High — routing rules rot as tools change |
| Multi-agent split by domain | Low per sub-agent | Good within domains, poor across them | Handoff overhead | High — orchestration and shared state |
| Tool Search | Roughly flat regardless of catalog size | Depends on retrieval quality | One extra round trip | Low — index is maintained for you |
| Tool Search plus pinning | Flat, plus pinned schemas | Best available: hot path guaranteed, tail retrieved | Only on the tail | Low |
Pinning is the part people skip and shouldn't. Foundry lets you pin critical tools so they bypass the search round-trip entirely, add context describing how your team actually thinks about a tool, and auto-pin frequently used ones. In practice a handful of tools account for most calls; pin those, retrieve everything else, and you get flat token cost without paying retrieval latency on the common path.
Two commands scaffold a hosted agent with a toolbox attached.
mkdir my-toolbox-agent && cd my-toolbox-agent
azd ai agent init \
-m "https://github.com/microsoft-foundry/foundry-samples/blob/main/samples/python/hosted-agents/agent-framework/responses/04-foundry-toolbox/agent.manifest.yaml" \
--src src/toolbox-agent
Then create the toolbox from the sample's descriptor:
azd ai toolbox create my-toolbox --from-file ./src/toolbox-agent/toolbox.yaml
That prints a versioned MCP endpoint, which is what your agent binds to:
https://<account>.services.ai.azure.com/api/projects/<project>/toolboxes/my-toolbox/versions/1/mcp?api-version=v1
One gotcha worth flagging because it cost me twenty minutes: azd ai toolbox create
needs a local azd project and environment to run against, even when you pass --project-endpoint
explicitly. If you aren't starting from azd ai agent init
, run azd init --minimal
first and set the endpoint into the environment:
azd init --minimal
azd env set FOUNDRY_PROJECT_ENDPOINT https://<account>.services.ai.azure.com/api/projects/<project>
Your toolbox.yaml
is where the catalog and its search behaviour are declared. The shape below is illustrative — preview schemas move, so check the current sample before copying:
name: my-toolbox
description: Order operations, customer lookup, and fulfilment tools
toolSearch:
enabled: true
pinned:
- get_order_status
- search_customers
autoPin:
enabled: true
minCallsPerWindow: 25
tools:
- name: get_order_status
source: mcp
server: orders-mcp
searchContext: >
Look up the current fulfilment state of a single order. Use when the
user mentions an order number, tracking number, "where is my package",
or asks whether something shipped.
- name: issue_refund
source: mcp
server: billing-mcp
searchContext: >
Issue a full or partial refund against a completed order. Requires an
order ID and an amount. Do not use for cancellations of unshipped
orders — use cancel_order instead.
That last searchContext
is doing real work. Retrieval quality is the entire ballgame with Tool Search, and retrieval runs against your descriptions. The explicit negative — don't use this one, use that one — is the single highest-value thing you can write there, because near-duplicates are precisely where selection fails.
The reason this topic is worth writing about rather than just enabling is that the win is measurable, and the size of the win depends entirely on your catalog. Wire up tracing before you change anything and get a baseline.
pip install azure-ai-projects azure-identity opentelemetry-sdk azure-core-tracing-opentelemetry
python
from azure.core.settings import settings
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter
from azure.ai.projects.telemetry import AIProjectInstrumentor
settings.tracing_implementation = "opentelemetry"
span_exporter = ConsoleSpanExporter()
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(span_exporter))
trace.set_tracer_provider(tracer_provider)
AIProjectInstrumentor().instrument()
Set AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING=true
before running, and every turn — prompts, tool calls, model responses — surfaces as a span with token usage attached. Swap ConsoleSpanExporter
for the Azure Monitor exporter and the same spans land in the Foundry portal's Observability tab.
Now run the comparison. The harness below is deliberately dumb: a fixed task set, both configurations, aggregate the spans.
import os
import json
from dataclasses import dataclass, field
from statistics import mean
@dataclass
class RunResult:
config: str
input_tokens: list = field(default_factory=list)
turns: list = field(default_factory=list)
correct_tool: list = field(default_factory=list)
TASKS = [
("Where is order 88231?", "get_order_status"),
("Refund the second item on order 88231", "issue_refund"),
("Cancel order 90114, it hasn't shipped", "cancel_order"),
("Which customers in Ohio ordered twice?", "search_customers"),
("Send the June invoice to billing@acme.com", "email_invoice"),
]
def score_run(spans, expected_tool):
"""Pull token usage and the actually-invoked tool out of collected spans."""
input_tokens = sum(
s.attributes.get("gen_ai.usage.input_tokens", 0) for s in spans
)
invoked = [
s.attributes.get("gen_ai.tool.name")
for s in spans
if s.attributes.get("gen_ai.tool.name")
]
resolved = [
json.loads(s.attributes["gen_ai.tool.arguments"]).get("name", n)
if n == "call_tool" else n
for s, n in zip(spans, invoked)
]
return input_tokens, len(invoked), expected_tool in resolved
def report(results):
for r in results:
print(f"\n{r.config}")
print(f" mean input tokens/task : {mean(r.input_tokens):>8,.0f}")
print(f" mean turns/task : {mean(r.turns):>8.1f}")
print(f" tool selection accuracy: {mean(r.correct_tool):>8.1%}")
Run it against a flat catalog, then against the same catalog with Tool Search on, then again with your top five tools pinned. Three numbers, one table, and you'll know whether this is worth doing for your catalog rather than for a catalog in a blog post.
Being straight about the limits, because the failure modes are real:
Small catalogs get worse, not better. Under about thirty tools, you've added a round-trip and a retrieval failure mode to save tokens you weren't spending. Don't.
Retrieval misses are silent and confusing. When a flat-list agent picks the wrong tool, the trace shows it considering the right one. When Tool Search never surfaces the right tool, the trace shows an agent that behaved reasonably given what it was handed. Debugging shifts from "why did it choose badly" to "why wasn't this in the candidate set," which is a different skill and a less obvious one.
Latency moves to the wrong place. The extra round-trip lands at the start of a turn, before any useful work. For a background agent that's free. For anything a human is watching, pin aggressively.
Your descriptions are now load-bearing infrastructure. A vague searchContext
used to cost you occasional bad tool calls. Now it costs you tools that are functionally invisible. Budget review time for them the way you'd budget it for an API contract.
Strip Foundry out of this and the principle stands: anything you send on every turn should earn its place on every turn. Tool schemas were the first thing to hit this wall because they grow with integration count, but the same audit applies to system prompts that accumulated instructions nobody has read in six months, few-shot examples kept for a model version you no longer run, and retrieved context that's pasted in wholesale because trimming it was somebody's Q3 task.
Tool Search is a good implementation of a boring idea — retrieve instead of broadcast. The reason to adopt it isn't that it's new. It's that you can measure the difference in an afternoon, and the number is usually larger than you expect.
Preview APIs in this area are moving quickly — the azd commands and tracing setup above are current as of the June 2026 Foundry release, but verify schema-level details against the current samples before shipping.