I was going through Grok Build’s open-source codebase and found an interesting pattern in how it handles MCP tool discovery. Instead of injecting all tool schemas into the prompt, it uses BM25 to search a hidden tool catalog on demand. This post walks through what that means, why it matters, and how to build it yourself.
Most MCP harnesses today work like this: on startup, connect all servers, fetch every tool’s JSON schema, and inject all of it into the system prompt. The model now has full visibility into every tool available.
That works fine with 5 tools. It breaks down at 50, and is impractical at 200+.
Two things go wrong as the tool list grows:
Token cost. Every schema gets injected on every turn, whether the agent needs that tool or not. A single tool schema with good documentation can be 200–400 tokens. Multiply by 100 tools and you’re burning 20,000–40,000 tokens per turn just on tool definitions.
KV cache instability. LLMs cache the prefix of the prompt for reuse across turns. If the system prompt changes — because a new MCP server connected and added tools — the cache is invalidated. You pay full recomputation cost. Garry Tan flagged this exact issue earlier this year as a core reason “MCP sucks” at scale.
Grok Build’s answer to both problems is the same: don’t put tool schemas in the prompt at all. Put a search interface there instead.
Only two tools are exposed to the model in the system prompt:
search_tool(query) — searches the hidden tool catalog and returns a short list of candidates with their names and descriptions.
use_tool(tool_name, arguments) — dispatches the actual MCP call for the selected tool.
Every connected MCP tool’s schema lives in a hidden catalog. The model never sees schemas upfront. When the agent needs a tool, it calls search_tool with a natural-language query. The harness runs BM25 over tool names, server names, descriptions, and parameter names, and returns the top matches. The agent picks one, and only at that point does the harness hand over that tool’s full JSON schema — immediately before the invocation call.
The system prompt stays constant regardless of how many MCP servers are connected. Adding a new server updates the BM25 index, not the prompt. The KV cache stays valid.
BM25 (Best Matching 25) is a classical keyword-ranking algorithm. It takes a query and a set of text documents, and scores each document based on how relevant it is to the query.
It scores using three ideas:
Term frequency — if the query word appears more often in a document, that document scores higher. But with diminishing returns: going from 1 to 2 occurrences matters, going from 10 to 11 barely does.
Inverse document frequency — rare words across the catalog matter more than common ones. If every tool description contains the word “data” but only one contains “telemetry”, then a query for “telemetry” should rank that one tool very high.
Length normalization — a short, focused description matching your query is ranked equally or better than a long one that happens to include the same words.
For tool discovery, this works well. Tool names, server names, and descriptions use specific technical vocabulary. When someone searches “create github issue”, the tool create_issue in the github server wins clearly. There’s no ambiguity that needs embeddings to resolve.
Assume you have three MCP servers connected: github, postgres, and slack. Each has a few tools. Here’s the full implementation to index and search them with BM25.
BM25 works on plain text. For each tool, build a single string from its server name, tool name, description, and parameter names and descriptions:
function toolToDocument(tool) { const params = Object.entries(tool.inputSchema.properties ?? {}) .map(([name, s]) => `${name} ${s.description ?? ""}`).join(" "); return [ `server ${tool.server}`, `tool ${tool.name}`, tool.name.replaceAll("_", " "), // "create_issue" -> "create issue" tool.description, params, ].join(" ");}
The replaceAll(“_”, “ “) on the tool name is a small but important detail. People write queries in natural language. Tool names use snake_case. You want both to match.
Step 2 — Build the BM25 index
This is a complete, dependency-free BM25 implementation. No npm packages needed:
class BM25 { docs = []; docFreq = new Map(); avgDocLength = 0; constructor(k1 = 1.2, b = 0.75) { this.k1 = k1; this.b = b; } tokenize(text) { return text .replace(/([a-z0-9])([A-Z])/g, "$1 $2") // split camelCase .replace(/[_./:-]/g, " ") // split snake_case .toLowerCase().match(/[a-z0-9]+/g) ?? []; } index(documents) { for (const [id, doc] of documents.entries()) { const tokens = this.tokenize(doc); const tf = new Map(); for (const t of tokens) tf.set(t, (tf.get(t) ?? 0) + 1); this.docs.push({ id, tf, length: tokens.length }); for (const t of tf.keys()) this.docFreq.set(t, (this.docFreq.get(t) ?? 0) + 1); } this.avgDocLength = this.docs.reduce((s, d) => s + d.length, 0) / this.docs.length; } search(query, limit = 5) { const terms = [...new Set(this.tokenize(query))]; const N = this.docs.length; return this.docs.map((doc) => { let score = 0; for (const t of terms) { const f = doc.tf.get(t) ?? 0; if (!f) continue; const df = this.docFreq.get(t) ?? 0; const idf = Math.log(1 + (N - df + 0.5) / (df + 0.5)); const norm = f + this.k1 * (1 - this.b + this.b * doc.length / this.avgDocLength); score += idf * f * (this.k1 + 1) / norm; } return { id: doc.id, score }; }).filter(r => r.score > 0).sort((a, b) => b.score - a.score).slice(0, limit); }}
Step 3 — Wire it up and search
const bm25 = new BM25();bm25.index(tools.map(toolToDocument));// Query 1bm25.search("open a bug issue for missing token events");// -> github.create_issue (score: 2.41)// Query 2bm25.search("find Slack messages about deployment failure");// -> slack.search_messages (score: 2.05)// Query 3bm25.search("run SQL query check missing events");// -> postgres.run_readonly_query (score: 1.98)
Each query returns the right tool at the top. The model receives only the schema of the tool it selects — not all nine.
Here’s the full sequence inside the harness:
The key part is step 7. The schema is only handed to the model at the moment it’s about to be used. This is why Grok Build says the model “never has to guess parameters” — it gets the authoritative schema right before the call.
Where BM25 Falls Short
BM25 is lexical. It matches on words, not meaning. If a user asks to “log a ticket” but your tool is called create_issue, BM25 may not rank it well because the words don’t overlap.
For most MCP use cases, this is acceptable. The agent writes the search query itself, and agents tend to use tool-adjacent vocabulary. But if you see retrieval failures in your logs, the fix is to layer on top:
Exact name match first — if the query matches a qualified tool name directly, skip BM25.
BM25 second — handles the majority of keyword-overlapping queries.
Embedding / hybrid third — add only when you have data showing BM25 is missing things.
Grok Build itself describes the search as “deliberately simple.” Start simple. Optimize when you have signal.
The core idea here isn’t BM25 specifically. It’s the design decision to treat the tool catalog as a database rather than a prompt manifest.
You don’t load every row of a database into memory before running a query. You query for what you need. MCP tool catalogs should work the same way. BM25 is just the right implementation for this scale — fast, accurate for keyword-rich technical vocabulary, no dependencies, no external API calls.
Build the index. Keep schemas hidden. Return them only at invocation time. Your agent’s system prompt stays stable, your KV cache stays warm, and your token costs become predictable.
If the patterns in this post are relevant to your workflow, TokenTelemetry is worth trying. It’s a 100% local, open-source dashboard that tracks token usage across Claude Code, Gemini CLI, Codex, Antigravity CLI, Hermes agent, and 9+ other coding agents — sessions, tool calls, reasoning steps, and prompt cache hit rates, all in one place.
The reason to use it isn’t to count tokens. It’s to actually see what your agent is spending context on. You can answer questions like: how much of my system prompt is tool schemas? Which tools get called most? Where does context go during a long agentic session?
If you’re building or optimizing an agent harness, those numbers change how you design. Install it, point it at your agent, and let it run for a day. The data is more useful than any benchmark.
Tokentelemetry: https://tokentelemetry.com/docs/
Grok Build: https://github.com/xai-org/grok-build
Grok Build Uses BM25 to Give AI Agents a Searchable Toolbox — And It’s a Pattern Worth Stealing was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.