The Complete Guide to Tool Selection in AI Agents

Agent accuracy degrades as tool catalogs grow beyond 10-15 tools, with research showing retrieval-based selection triples accuracy from 13.62% to 43.13% while cutting prompt tokens by over half. Six techniques—gating, retrieval, routing, planning, fallback logic, and benchmarking—can maintain tool selection accuracy at scale without requiring larger models.

In this article, you will learn why agent accuracy degrades as a tool catalog grows, and six practical techniques for keeping tool selection accurate and efficient at scale. Topics we will cover include: - Why adding more tools to an agent causes tool hallucination and accuracy loss, not just slower responses. - How gating, retrieval, routing, and planning each narrow down what the model sees before it has to choose a tool. - How to build fallback logic and a benchmark harness so you can measure whether any of these fixes actually worked. None of this requires a bigger model, just a smarter view of what the model sees before it acts. Introduction You build an agent with five tools. It works flawlessly in the demo. Three months later, it has 40 file operations, CRM access, Slack, a calendar, and three different search APIs you bolted on for different teams. The same agent that nailed every demo now calls the wrong tool, hallucinates parameters borrowed from a different tool’s schema, or stalls mid-task waiting on a call that should never have been made. Nothing about the model changed. The tool list did. This is not an edge case you’ll eventually run into. It’s the default trajectory of every agent that ships and then grows. Research analyzing MCP tool descriptions across the ecosystem has found that a high number contain at least one quality issue, and production benchmarks show agent accuracy degrading measurably once tool counts pass roughly 10 to 15. The RAG-MCP paper https://arxiv.org/abs/2505.03275 , published in May 2025, put hard numbers on the fix: retrieval-based tool selection more than tripled tool selection accuracy from 13.62% to 43.13% while cutting prompt tokens by over half on the same benchmark tasks. Tool selection isn’t a minor implementation detail you patch later. It’s the architectural decision that determines whether an agent survives contact with a real tool catalog. This guide covers six techniques that solve it, in the order you’d actually deploy them: gating, retrieval, routing, planning, fallback logic, and the benchmark that tells you whether any of it worked. Why Tool Selection Breaks at Scale Every tool definition — its name, description, and parameter schema — gets sent to the model on every single request, whether that tool gets used or not. With 50-plus tools, this can consume 5 to 7% of the model’s context before the user’s actual message arrives, crowding out the conversation history and reasoning space the task actually needs. The “ lost in the middle ” effect compounds this. Models recall information at the start and end of a context window far more reliably than information buried in the middle. With dozens of near-identical tool definitions stacked in sequence, the one tool that’s actually right for the job often sits exactly in that dead zone, overlooked not because the model can’t reason about it, but because attention is structurally pulled elsewhere. The second failure mode is worse: tool hallucination. When an LLM’s attention spreads across too many similar-sounding tools, it either invents tool names that don’t exist or calls the correct tool while filling in arguments borrowed from a different tool’s schema. This is a hard failure. There’s no “ slightly wrong ” way to call a nonexistent function. OpenAI documents a hard ceiling of 128 tools per agent https://community.openai.com/t/maximum-amount-of-tools-for-the-bot-to-use/665720 , but real degradation shows up well before that limit; most production teams see accuracy drop noticeably once they cross 15 to 20 tools in active rotation. The fix isn’t a bigger context window. It’s controlling what the model sees in the first place. Gating: Deciding Whether a Tool Is Needed at All Before you optimize which tool to pick, ask a cheaper question first: does this turn need a tool at all? A meaningful fraction of agent turns are purely conversational: “ thanks ,” “ what do you mean by that ,” a follow-up clarification. Running full retrieval and tool-selection reasoning on every single turn means paying the full agentic overhead even when the answer is “ no tool needed .” A gate is a fast, cheap classifier — sometimes a small model call, sometimes just pattern matching — that runs before anything expensive does. gate.py Prerequisites: none beyond Python's standard library re Run: python gate.py import re CONVERSATIONAL PATTERNS = r"^\s thanks|thank you|thx|ok|okay|cool|got it|sounds good|sure|great \b", r"^\s hi|hello|hey|good morning|good evening \b", r"^\s what do you mean\b", r"^\s can you clarify|explain that \b", ACTION KEYWORDS = "send", "create", "search", "find", "look up", "schedule", "book", "read", "write", "query", "summarize", "translate", "check", def gate query: str - dict: """ Cheap pre-filter that decides whether the full tool-selection pipeline needs to run at all. Short-circuits conversational turns before retrieval, routing, or planning ever fires. """ q lower = query.strip .lower Tier 1: regex match against known conversational patterns -- near-zero cost for pattern in CONVERSATIONAL PATTERNS: if re.match pattern, q lower : return {"tool needed": False, "reason": "conversational pattern", "tier": 1} Tier 2: if there's no action verb and the message is short, likely no tool needed has action keyword = any kw in q lower for kw in ACTION KEYWORDS if not has action keyword and len q lower.split < 5: return {"tool needed": False, "reason": "short with no action keyword", "tier": 2} return {"tool needed": True, "reason": "action keyword or long query", "tier": 2} if name == " main ": test queries = "thanks ", "What's the weather like in Lagos today?", "ok", "Can you send an email to the sales team about the delay?", for q in test queries: result = gate q print f"'{q}' - tool needed={result 'tool needed' } {result 'reason' } " 12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849 gate.py Prerequisites: none beyond Python's standard library re Run: python gate.py import re CONVERSATIONAL PATTERNS = r"^\s thanks|thank you|thx|ok|okay|cool|got it|sounds good|sure|great \b", r"^\s hi|hello|hey|good morning|good evening \b", r"^\s what do you mean\b", r"^\s can you clarify|explain that \b", ACTION KEYWORDS = "send", "create", "search", "find", "look up", "schedule", "book", "read", "write", "query", "summarize", "translate", "check", def gate query: str - dict: """ Cheap pre-filter that decides whether the full tool-selection pipeline needs to run at all. Short-circuits conversational turns before retrieval, routing, or planning ever fires. """ q lower = query.strip .lower Tier 1: regex match against known conversational patterns -- near-zero cost for pattern in CONVERSATIONAL PATTERNS: if re.match pattern, q lower : return {"tool needed": False, "reason": "conversational pattern", "tier": 1} Tier 2: if there's no action verb and the message is short, likely no tool needed has action keyword = any kw in q lower for kw in ACTION KEYWORDS if not has action keyword and len q lower.split < 5: return {"tool needed": False, "reason": "short with no action keyword", "tier": 2} return {"tool needed": True, "reason": "action keyword or long query", "tier": 2} if name == " main ": test queries = "thanks ", "What's the weather like in Lagos today?", "ok", "Can you send an email to the sales team about the delay?", for q in test queries: result = gate q print f"'{q}' - tool needed={result 'tool needed' } {result 'reason' } " How to run no dependencies required : python gate.py 1 python gate.py This costs almost nothing and catches a meaningful share of turns before they reach the expensive part of the pipeline. The threshold for “is this worth building” is low: if even 20–30% of your turns are conversational, gating pays for itself immediately in both latency and token cost. Retrieval-Based Tool Selection This is the technique with the strongest published evidence behind it. Instead of sending every tool definition on every call, you index tool descriptions in a vector store, embed the incoming query, retrieve only the top-K most relevant tools, and send just those to the model. The RAG-MCP framework https://arxiv.org/abs/2505.03275 is the reference implementation of this idea, using semantic retrieval to identify the most relevant MCP tools for a query before the LLM ever sees the full catalog. The reported numbers are not subtle: tool selection accuracy rose from 13.62% with the full catalog exposed to 43.13% with retrieval-filtered selection, more than tripling accuracy, while cutting prompt tokens by over 50% on the same benchmark tasks. retriever.py Prerequisites: pip install sentence-transformers faiss-cpu numpy Run: python retriever.py import numpy as np from sentence transformers import SentenceTransformer import faiss TOOL CATALOG = {"name": "search web", "description": "Search the web for current information on any topic"}, {"name": "read file", "description": "Read the contents of a file given its path"}, {"name": "write file", "description": "Write or overwrite content to a file at a given path"}, {"name": "send email", "description": "Send an email to a recipient with subject and body"}, {"name": "create calendar event", "description": "Create a new calendar event with a title, date, and time"}, {"name": "query database", "description": "Run a SQL query against the company database"}, {"name": "list github issues", "description": "List open issues in a GitHub repository"}, {"name": "create github pr", "description": "Create a pull request on a GitHub repository"}, {"name": "send slack message", "description": "Send a message to a Slack channel or user"}, {"name": "get weather", "description": "Get current weather conditions for a city"}, {"name": "translate text", "description": "Translate text from one language to another"}, {"name": "summarize document", "description": "Summarize a long document into key points"}, {"name": "lookup stock price", "description": "Get the current stock price for a ticker symbol"}, {"name": "book flight", "description": "Search and book a flight between two cities"}, {"name": "create invoice", "description": "Generate an invoice for a customer with line items"}, class ToolRetriever: """ Embeds tool descriptions once at startup and indexes them in FAISS. At runtime, embeds the incoming query and returns only the top-K most relevant tools -- not the full catalog. """ def init self, tools: list dict , model name: str = "all-MiniLM-L6-v2" : self.tools = tools self.model = SentenceTransformer model name descriptions = f"{t 'name' }: {t 'description' }" for t in tools embeddings = self.model.encode descriptions, normalize embeddings=True IndexFlatIP = inner product search, which equals cosine similarity when vectors are normalized -- the standard setup for this use case. self.index = faiss.IndexFlatIP embeddings.shape 1 self.index.add np.array embeddings, dtype=np.float32 def retrieve self, query: str, top k: int = 3 - list dict : query emb = self.model.encode query , normalize embeddings=True scores, indices = self.index.search np.array query emb, dtype=np.float32 , top k return { self.tools idx , "score": float score } for score, idx in zip scores 0 , indices 0 if name == " main ": retriever = ToolRetriever TOOL CATALOG queries = "What's the weather like in Lagos today?", "Can you check if there are any open bugs in our repo?", "Send a message to the engineering channel about the deploy", for q in queries: results = retriever.retrieve q, top k=3 print f"\nQuery: '{q}'" for r in results: print f" {r 'name' } score={r 'score' :.3f} " 12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364 retriever.py Prerequisites: pip install sentence-transformers faiss-cpu numpy Run: python retriever.py import numpy as npfrom sentence transformers import SentenceTransformerimport faiss TOOL CATALOG = {"name": "search web", "description": "Search the web for current information on any topic"}, {"name": "read file", "description": "Read the contents of a file given its path"}, {"name": "write file", "description": "Write or overwrite content to a file at a given path"}, {"name": "send email", "description": "Send an email to a recipient with subject and body"}, {"name": "create calendar event", "description": "Create a new calendar event with a title, date, and time"}, {"name": "query database", "description": "Run a SQL query against the company database"}, {"name": "list github issues", "description": "List open issues in a GitHub repository"}, {"name": "create github pr", "description": "Create a pull request on a GitHub repository"}, {"name": "send slack message", "description": "Send a message to a Slack channel or user"}, {"name": "get weather", "description": "Get current weather conditions for a city"}, {"name": "translate text", "description": "Translate text from one language to another"}, {"name": "summarize document", "description": "Summarize a long document into key points"}, {"name": "lookup stock price", "description": "Get the current stock price for a ticker symbol"}, {"name": "book flight", "description": "Search and book a flight between two cities"}, {"name": "create invoice", "description": "Generate an invoice for a customer with line items"}, class ToolRetriever: """ Embeds tool descriptions once at startup and indexes them in FAISS. At runtime, embeds the incoming query and returns only the top-K most relevant tools -- not the full catalog. """ def init self, tools: list dict , model name: str = "all-MiniLM-L6-v2" : self.tools = tools self.model = SentenceTransformer model name descriptions = f"{t 'name' }: {t 'description' }" for t in tools embeddings = self.model.encode descriptions, normalize embeddings=True IndexFlatIP = inner product search, which equals cosine similarity when vectors are normalized -- the standard setup for this use case. self.index = faiss.IndexFlatIP embeddings.shape 1 self.index.add np.array embeddings, dtype=np.float32 def retrieve self, query: str, top k: int = 3 - list dict : query emb = self.model.encode query , normalize embeddings=True scores, indices = self.index.search np.array query emb, dtype=np.float32 , top k return { self.tools idx , "score": float score } for score, idx in zip scores 0 , indices 0 if name == " main ": retriever = ToolRetriever TOOL CATALOG queries = "What's the weather like in Lagos today?", "Can you check if there are any open bugs in our repo?", "Send a message to the engineering channel about the deploy", for q in queries: results = retriever.retrieve q, top k=3 print f"\nQuery: '{q}'" for r in results: print f" {r 'name' } score={r 'score' :.3f} " How to run: pip install sentence-transformers faiss-cpu numpy python retriever.py 12 pip install sentence-transformers faiss-cpu numpypython retriever.py Only the top-3 tools out of a 15-tool catalog get sent to the model per query, an 80% reduction in tool definitions on every call, and the accuracy lift compounds because the model is now choosing between a handful of genuinely relevant candidates instead of scanning past a dozen near-misses. Semantic Routing Routing is retrieval’s lighter cousin, and it fits a different shape of problem. Retrieval answers “ which specific tool ” out of a flat list. Routing answers “ which toolbox ” — useful when your tools cluster naturally into categories data, communication, scheduling and you want to load only the relevant category’s tools rather than re-ranking the entire catalog every time. router.py Prerequisites: pip install scikit-learn numpy Run: python router.py import numpy as np from sklearn.feature extraction.text import TfidfVectorizer from sklearn.metrics.pairwise import cosine similarity CATEGORIES = { "data": "query the database", "read a file", "write data to storage", "run a SQL query" , "communication": "send an email", "post a slack message", "notify the team", "send a message" , "scheduling": "create a calendar event", "book a meeting", "schedule an appointment" , } class SemanticRouter: """ Routes a query to a tool category using similarity against category centroid embeddings. Falls back to 'general' when no category clears the confidence threshold -- never guesses on a low-confidence match. """ def init self, categories: dict str, list str , confidence threshold: float = 0.15 : self.threshold = confidence threshold self.vectorizer = TfidfVectorizer all examples = ex for exs in categories.values for ex in exs self.vectorizer.fit all examples Build one centroid vector per category by averaging its example embeddings self.centroids = {} for cat, examples in categories.items : vecs = self.vectorizer.transform examples .toarray self.centroids cat = vecs.mean axis=0 def route self, query: str - dict: query vec = self.vectorizer.transform query .toarray 0 scores = { cat: float cosine similarity query vec , centroid 0 0 for cat, centroid in self.centroids.items } best cat = max scores, key=scores.get best score = scores best cat if best score < self.threshold: return {"category": "general", "confidence": best score} return {"category": best cat, "confidence": best score} if name == " main ": router = SemanticRouter CATEGORIES test queries = "Can you post a message in the sales Slack channel?", "I need to run a query against our production database", "Schedule a meeting with the design team for tomorrow", "asdkj qpwoe zxcv nonsense", for q in test queries: result = router.route q print f"'{q}' - {result 'category' } confidence={result 'confidence' :.3f} " 1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859 router.py Prerequisites: pip install scikit-learn numpy Run: python router.py import numpy as npfrom sklearn.feature extraction.text import TfidfVectorizerfrom sklearn.metrics.pairwise import cosine similarity CATEGORIES = { "data": "query the database", "read a file", "write data to storage", "run a SQL query" , "communication": "send an email", "post a slack message", "notify the team", "send a message" , "scheduling": "create a calendar event", "book a meeting", "schedule an appointment" ,} class SemanticRouter: """ Routes a query to a tool category using similarity against category centroid embeddings. Falls back to 'general' when no category clears the confidence threshold -- never guesses on a low-confidence match. """ def init self, categories: dict str, list str , confidence threshold: float = 0.15 : self.threshold = confidence threshold self.vectorizer = TfidfVectorizer all examples = ex for exs in categories.values for ex in exs self.vectorizer.fit all examples Build one centroid vector per category by averaging its example embeddings self.centroids = {} for cat, examples in categories.items : vecs = self.vectorizer.transform examples .toarray self.centroids cat = vecs.mean axis=0 def route self, query: str - dict: query vec = self.vectorizer.transform query .toarray 0 scores = { cat: float cosine similarity query vec , centroid 0 0 for cat, centroid in self.centroids.items } best cat = max scores, key=scores.get best score = scores best cat if best score < self.threshold: return {"category": "general", "confidence": best score} return {"category": best cat, "confidence": best score} if name == " main ": router = SemanticRouter CATEGORIES test queries = "Can you post a message in the sales Slack channel?", "I need to run a query against our production database", "Schedule a meeting with the design team for tomorrow", "asdkj qpwoe zxcv nonsense", for q in test queries: result = router.route q print f"'{q}' - {result 'category' } confidence={result 'confidence' :.3f} " How to run: pip install scikit-learn numpy python router.py 12 pip install scikit-learn numpypython router.py The fallback to “ general ” on the gibberish query matters as much as the correct routes do. A router that always picks something, even on a query it has no real signal for, is more dangerous than one that admits it doesn’t know. Planner-Based Tool Selection Retrieval and routing both answer “ what’s relevant to this single turn .” Multi-step tasks need something different: a sequence of tool calls planned upfront, with each step scoped to only the tools it specifically needs. This is the architecture that avoids what’s sometimes called the God Agent anti-pattern — a single agent holding 20 tools in context with no plan structure — where a failure anywhere corrupts the whole task. The pattern : ask the model to output a structured plan first, an ordered list of subtasks, each tagged with the capability it requires, before any tool executes. Then retrieve tools per step, scoped to that step’s tag. planner.py Prerequisites: none beyond Python's standard library json Run: python planner.py import json from dataclasses import dataclass @dataclass class PlanStep: step number: int description: str required capability: str def parse plan raw plan json: str - list PlanStep : """Parse a planner LLM's JSON output into structured PlanStep objects.""" data = json.loads raw plan json return PlanStep s "step number" , s "description" , s "required capability" for s in data "steps" Capability - tool-name mapping. In production this feeds the retriever from the previous section, scoped to only the tools tagged for this capability. CAPABILITY TOOLS = { "search": "search web", "query database" , "file io": "read file", "write file" , "communication": "send email", "send slack message" , "synthesis": "summarize document" , } def get scoped tools step: PlanStep - list str : """Return only the tools relevant to this step -- not the full catalog.""" return CAPABILITY TOOLS.get step.required capability, if name == " main ": This JSON would normally come from an LLM call asking it to decompose the task into steps, each tagged with the capability it needs. mock plan = json.dumps { "steps": {"step number": 1, "description": "Search for the latest sales report file", "required capability": "search"}, {"step number": 2, "description": "Read the contents of the report file", "required capability": "file io"}, {"step number": 3, "description": "Summarize the key findings", "required capability": "synthesis"}, {"step number": 4, "description": "Email the summary to the sales lead", "required capability": "communication"}, } plan = parse plan mock plan for step in plan: scoped = get scoped tools step print f"Step {step.step number}: {step.description}" print f" Capability: {step.required capability} - tools available: {scoped}" 12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152 planner.py Prerequisites: none beyond Python's standard library json Run: python planner.py import jsonfrom dataclasses import dataclass @dataclassclass PlanStep: step number: int description: str required capability: str def parse plan raw plan json: str - list PlanStep : """Parse a planner LLM's JSON output into structured PlanStep objects.""" data = json.loads raw plan json return PlanStep s "step number" , s "description" , s "required capability" for s in data "steps" Capability - tool-name mapping. In production this feeds the retriever from the previous section, scoped to only the tools tagged for this capability.CAPABILITY TOOLS = { "search": "search web", "query database" , "file io": "read file", "write file" , "communication": "send email", "send slack message" , "synthesis": "summarize document" ,} def get scoped tools step: PlanStep - list str : """Return only the tools relevant to this step -- not the full catalog.""" return CAPABILITY TOOLS.get step.required capability, if name == " main ": This JSON would normally come from an LLM call asking it to decompose the task into steps, each tagged with the capability it needs. mock plan = json.dumps { "steps": {"step number": 1, "description": "Search for the latest sales report file", "required capability": "search"}, {"step number": 2, "description": "Read the contents of the report file", "required capability": "file io"}, {"step number": 3, "description": "Summarize the key findings", "required capability": "synthesis"}, {"step number": 4, "description": "Email the summary to the sales lead", "required capability": "communication"}, } plan = parse plan mock plan for step in plan: scoped = get scoped tools step print f"Step {step.step number}: {step.description}" print f" Capability: {step.required capability} - tools available: {scoped}" How to run no dependencies required : python planner.py 1 python planner.py Each step in this example sees one or two tools, never the full set. That’s the actual mechanism behind why planning helps: it’s not that the model reasons better when it has a plan; it’s that the plan lets you legitimately narrow the tool list per step, which is the same lever retrieval pulls, applied at a finer grain. Fallback Logic Retrieval and routing both fail sometimes, not because the architecture is wrong, but because real queries are ambiguous, underspecified, or genuinely outside the tool catalog’s coverage. What you do when the top match’s confidence is low determines whether your agent degrades gracefully or starts guessing. A three-tier fallback chain handles this without resorting to a try/except that just crashes the conversation: resolve directly when confidence is high, retry with a reformulated query when it isn’t, and escalate to an explicit clarification request rather than forcing a tool call when even the retry comes up short. fallback.py Prerequisites: pip install scikit-learn numpy Run: python fallback.py import numpy as np from sklearn.feature extraction.text import TfidfVectorizer from sklearn.metrics.pairwise import cosine similarity TOOL CATALOG = {"name": "search web", "description": "Search the web for current information on any topic"}, {"name": "get weather", "description": "Get the current weather forecast for a city"}, {"name": "send email", "description": "Send an email to a recipient with subject and body"}, {"name": "list github issues", "description": "List open issues and bugs in a GitHub repository"}, class RetrieverWithFallback: """ Wraps retrieval with a three-tier fallback chain: 1. High confidence - use the top result directly 2. Low confidence - retry with a reformulated query 3. Still low - escalate to a clarification request, never guess """ def init self, tools, confidence threshold: float = 0.12 : self.tools = tools self.threshold = confidence threshold self.vectorizer = TfidfVectorizer descs = f"{t 'name' }: {t 'description' }" for t in tools self.tool vectors = self.vectorizer.fit transform descs def raw retrieve self, query: str : query vec = self.vectorizer.transform query sims = cosine similarity query vec, self.tool vectors 0 top idx = int np.argmax sims return self.tools top idx , float sims top idx def retrieve with fallback self, query: str - dict: tool, score = self. raw retrieve query if score = self.threshold: return {"status": "resolved", "tool": tool "name" , "confidence": score, "attempts": 1} Reformulate by stripping filler words. In production, this step would be an LLM call asking it to restate the query in terms of intent/capability. reformulated = query.replace "can you", "" .replace "please", "" .replace "?", "" .strip tool2, score2 = self. raw retrieve reformulated if score2 = self.threshold: return {"status": "resolved", "tool": tool2 "name" , "confidence": score2, "attempts": 2} return { "status": "escalated", "tool": None, "confidence": max score, score2 , "attempts": 2, "clarification request": f"I'm not confident which tool fits '{query}'. " f"Could you clarify what you'd like me to do?" , } if name == " main ": retriever = RetrieverWithFallback TOOL CATALOG for q in "What's the weather forecast in Lagos?", "xyzzy plugh random nonsense" : result = retriever.retrieve with fallback q print f"Query: '{q}'" print f" Status: {result 'status' }" if result "status" == "resolved": print f" Tool: {result 'tool' } confidence={result 'confidence' :.3f} " else: print f" {result 'clarification request' }" 1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768 fallback.py Prerequisites: pip install scikit-learn numpy Run: python fallback.py import numpy as npfrom sklearn.feature extraction.text import TfidfVectorizerfrom sklearn.metrics.pairwise import cosine similarity TOOL CATALOG = {"name": "search web", "description": "Search the web for current information on any topic"}, {"name": "get weather", "description": "Get the current weather forecast for a city"}, {"name": "send email", "description": "Send an email to a recipient with subject and body"}, {"name": "list github issues", "description": "List open issues and bugs in a GitHub repository"}, class RetrieverWithFallback: """ Wraps retrieval with a three-tier fallback chain: 1. High confidence - use the top result directly 2. Low confidence - retry with a reformulated query 3. Still low - escalate to a clarification request, never guess """ def init self, tools, confidence threshold: float = 0.12 : self.tools = tools self.threshold = confidence threshold self.vectorizer = TfidfVectorizer descs = f"{t 'name' }: {t 'description' }" for t in tools self.tool vectors = self.vectorizer.fit transform descs def raw retrieve self, query: str : query vec = self.vectorizer.transform query sims = cosine similarity query vec, self.tool vectors 0 top idx = int np.argmax sims return self.tools top idx , float sims top idx def retrieve with fallback self, query: str - dict: tool, score = self. raw retrieve query if score = self.threshold: return {"status": "resolved", "tool": tool "name" , "confidence": score, "attempts": 1} Reformulate by stripping filler words. In production, this step would be an LLM call asking it to restate the query in terms of intent/capability. reformulated = query.replace "can you", "" .replace "please", "" .replace "?", "" .strip tool2, score2 = self. raw retrieve reformulated if score2 = self.threshold: return {"status": "resolved", "tool": tool2 "name" , "confidence": score2, "attempts": 2} return { "status": "escalated", "tool": None, "confidence": max score, score2 , "attempts": 2, "clarification request": f"I'm not confident which tool fits '{query}'. " f"Could you clarify what you'd like me to do?" , } if name == " main ": retriever = RetrieverWithFallback TOOL CATALOG for q in "What's the weather forecast in Lagos?", "xyzzy plugh random nonsense" : result = retriever.retrieve with fallback q print f"Query: '{q}'" print f" Status: {result 'status' }" if result "status" == "resolved": print f" Tool: {result 'tool' } confidence={result 'confidence' :.3f} " else: print f" {result 'clarification request' }" How to run: pip install scikit-learn numpy python fallback.py 12 pip install scikit-learn numpypython fallback.py The escalation path is the one most teams skip when they first build this, and it’s the one that matters most in production. A confidently wrong tool call is worse than a system that asks, “ I’m not sure, could you clarify? ” The second failure mode is recoverable in one turn. The first one usually isn’t. Benchmarking Your Tool Selection System Everything above is a hypothesis until you measure it. The methodology is straightforward: build a labeled set of query, correct tool pairs, run your pipeline against it, and measure accuracy, token cost, and latency, comparing your filtered pipeline against the naive full-catalog baseline. MCPToolBench++ https://arxiv.org/pdf/2508.07575 , a large-scale benchmark built from over 4,000 real MCP servers across 40-plus categories, is the reference for how rigorously this should be structured at scale, but the core idea works at any size. benchmark.py Prerequisites: pip install scikit-learn numpy Run: python benchmark.py import time import numpy as np from sklearn.feature extraction.text import TfidfVectorizer from sklearn.metrics.pairwise import cosine similarity TOOL CATALOG = {"name": "search web", "description": "Search the web for current information on any topic"}, {"name": "read file", "description": "Read the contents of a file given its path"}, {"name": "write file", "description": "Write or overwrite content to a file at a given path"}, {"name": "send email", "description": "Send an email to a recipient with subject and body"}, {"name": "create calendar event", "description": "Create a new calendar event with a title and time"}, {"name": "query database", "description": "Run a SQL query against the company database"}, {"name": "list github issues", "description": "List open issues and bugs in a GitHub repository"}, {"name": "send slack message", "description": "Send a message to a Slack channel or user"}, {"name": "get weather", "description": "Get current weather conditions for a city"}, {"name": "book flight", "description": "Search and book a flight between two cities"}, Labeled benchmark set: query, expected tool . Build yours from real logged queries once you have production traffic -- this is a seed set. BENCHMARK SET = "What's the weather in Abuja right now?", "get weather" , "Send an email to the finance team", "send email" , "List the open issues on our main repo", "list github issues" , "Book me a flight from Lagos to London", "book flight" , "Query the database for last week's signups", "query database" , "Post an update in the team Slack channel", "send slack message" , "Search the web for the latest interest rates", "search web" , "Read the contents of config.yaml", "read file" , def estimate tokens text: str - int: """Rough token estimate 1 token ~ 4 characters -- good enough for relative comparison.""" return len text // 4 class BenchmarkHarness: """Runs a labeled query set through a retriever and reports accuracy, token cost, and latency.""" def init self, tools: list dict , top k: int = 3 : self.tools = tools self.top k = top k self.vectorizer = TfidfVectorizer descs = f"{t 'name' }: {t 'description' }" for t in tools self.tool vectors = self.vectorizer.fit transform descs self.full catalog tokens = sum estimate tokens d for d in descs def retrieve self, query: str, top k: int - list dict : query vec = self.vectorizer.transform query sims = cosine similarity query vec, self.tool vectors 0 top indices = np.argsort sims ::-1 :top k return self.tools i for i in top indices def run self, benchmark set: list tuple , use retrieval: bool = True - dict: correct, total tokens, latencies = 0, 0, for query, expected tool in benchmark set: t0 = time.perf counter if use retrieval: candidates = self. retrieve query, top k=self.top k tokens this query = sum estimate tokens f"{t 'name' }: {t 'description' }" for t in candidates else: Baseline: send the full, unfiltered catalog every time candidates = self.tools tokens this query = self.full catalog tokens if expected tool in c "name" for c in candidates : correct += 1 total tokens += tokens this query latencies.append time.perf counter - t0 n = len benchmark set latencies sorted = sorted latencies return { "accuracy": round correct / n, 4 , "avg tokens": round total tokens / n, 1 , "p50 latency ms": round latencies sorted len latencies sorted // 2 1000, 3 , "p95 latency ms": round latencies sorted int len latencies sorted 0.95 1000, 3 , } if name == " main ": harness = BenchmarkHarness TOOL CATALOG, top k=3 baseline = harness.run BENCHMARK SET, use retrieval=False retrieval = harness.run BENCHMARK SET, use retrieval=True print "Baseline full catalog every time :" print f" Accuracy: {baseline 'accuracy' 100:.1f}%" print f" Avg tokens/query: {baseline 'avg tokens' }" print "\nRetrieval-filtered top-3 :" print f" Accuracy: {retrieval 'accuracy' 100:.1f}%" print f" Avg tokens/query: {retrieval 'avg tokens' }" reduction = 1 - retrieval "avg tokens" / baseline "avg tokens" 100 print f"\nToken reduction with retrieval: {reduction:.1f}%" 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103 benchmark.py Prerequisites: pip install scikit-learn numpy Run: python benchmark.py import timeimport numpy as npfrom sklearn.feature extraction.text import TfidfVectorizerfrom sklearn.metrics.pairwise import cosine similarity TOOL CATALOG = {"name": "search web", "description": "Search the web for current information on any topic"}, {"name": "read file", "description": "Read the contents of a file given its path"}, {"name": "write file", "description": "Write or overwrite content to a file at a given path"}, {"name": "send email", "description": "Send an email to a recipient with subject and body"}, {"name": "create calendar event", "description": "Create a new calendar event with a title and time"}, {"name": "query database", "description": "Run a SQL query against the company database"}, {"name": "list github issues", "description": "List open issues and bugs in a GitHub repository"}, {"name": "send slack message", "description": "Send a message to a Slack channel or user"}, {"name": "get weather", "description": "Get current weather conditions for a city"}, {"name": "book flight", "description": "Search and book a flight between two cities"}, Labeled benchmark set: query, expected tool . Build yours from real logged queries once you have production traffic -- this is a seed set.BENCHMARK SET = "What's the weather in Abuja right now?", "get weather" , "Send an email to the finance team", "send email" , "List the open issues on our main repo", "list github issues" , "Book me a flight from Lagos to London", "book flight" , "Query the database for last week's signups", "query database" , "Post an update in the team Slack channel", "send slack message" , "Search the web for the latest interest rates", "search web" , "Read the contents of config.yaml", "read file" , def estimate tokens text: str - int: """Rough token estimate 1 token ~ 4 characters -- good enough for relative comparison.""" return len text // 4 class BenchmarkHarness: """Runs a labeled query set through a retriever and reports accuracy, token cost, and latency.""" def init self, tools: list dict , top k: int = 3 : self.tools = tools self.top k = top k self.vectorizer = TfidfVectorizer descs = f"{t 'name' }: {t 'description' }" for t in tools self.tool vectors = self.vectorizer.fit transform descs self.full catalog tokens = sum estimate tokens d for d in descs def retrieve self, query: str, top k: int - list dict : query vec = self.vectorizer.transform query sims = cosine similarity query vec, self.tool vectors 0 top indices = np.argsort sims ::-1 :top k return self.tools i for i in top indices def run self, benchmark set: list tuple , use retrieval: bool = True - dict: correct, total tokens, latencies = 0, 0, for query, expected tool in benchmark set: t0 = time.perf counter if use retrieval: candidates = self. retrieve query, top k=self.top k tokens this query = sum estimate tokens f"{t 'name' }: {t 'description' }" for t in candidates else: Baseline: send the full, unfiltered catalog every time candidates = self.tools tokens this query = self.full catalog tokens if expected tool in c "name" for c in candidates : correct += 1 total tokens += tokens this query latencies.append time.perf counter - t0 n = len benchmark set latencies sorted = sorted latencies return { "accuracy": round correct / n, 4 , "avg tokens": round total tokens / n, 1 , "p50 latency ms": round latencies sorted len latencies sorted // 2 1000, 3 , "p95 latency ms": round latencies sorted int len latencies sorted 0.95 1000, 3 , } if name == " main ": harness = BenchmarkHarness TOOL CATALOG, top k=3 baseline = harness.run BENCHMARK SET, use retrieval=False retrieval = harness.run BENCHMARK SET, use retrieval=True print "Baseline full catalog every time :" print f" Accuracy: {baseline 'accuracy' 100:.1f}%" print f" Avg tokens/query: {baseline 'avg tokens' }" print "\nRetrieval-filtered top-3 :" print f" Accuracy: {retrieval 'accuracy' 100:.1f}%" print f" Avg tokens/query: {retrieval 'avg tokens' }" reduction = 1 - retrieval "avg tokens" / baseline "avg tokens" 100 print f"\nToken reduction with retrieval: {reduction:.1f}%" How to run: pip install scikit-learn numpy python benchmark.py 12 pip install scikit-learn numpypython benchmark.py On this 10-tool catalog with an 8-query benchmark set, retrieval-filtering held accuracy steady while cutting average tokens per query by roughly 70%. The exact numbers will shift with your catalog and query set, but the comparison structure is what matters: you now have a repeatable way to answer “did this change actually help” instead of relying on a handful of manual spot checks. Wrapping up These six techniques aren’t competing options; they’re layers. Gating filters out turns that need no tool at all, cheaply, before anything else runs. Retrieval or routing narrows the catalog down to what’s actually relevant for the turns that remain. Planning sequences of multi-step tasks so each step only sees the tools it needs. Fallback logic catches the cases where the first attempt doesn’t land cleanly. Benchmarking is how you know whether any of the above made a measurable difference, rather than just feeling better. The RAG-MCP result, with accuracy more than tripling and tokens cut by half, isn’t an outlier. It’s what happens predictably once you stop asking a model to read through a full phone book before every decision. None of these techniques requires a bigger model or a longer context window. They require treating the tool list itself as something to be designed, not just appended to. Resources :