Cutting Agent Token Costs in .NET: Tool Search, Output Gating and What the Numbers Really Say

A hands-on C# experiment using llama.cpp's llama-server with the Qwen2 tokenizer found that replacing a full MCP-style tool catalog with two meta tools (tool_search and call_tool) cut tool-definition tokens by 85.8% at 15 tools, 91.8% at 30 tools, 95.3% at 60 tools and 97.8% at 120 tools, with the meta-tool pair costing a flat 278 tokens against 1,952 to 12,617 for the full lists. The author reports that some savings reproduced, one technique made costs worse in the configuration commonly recommended, and another broke when an unnamed detail changed, and notes the test model used random weights so results measure prompt layout cost only, not whether a real model would select the correct tool. The post cites Microsoft's own benchmark, as reported by Nithin, of over 60% fewer input tokens at 50 tools and over 97% at 1,000 tools, measured against a prompt-cached baseline.

Every few days another post tells me I can cut my agent’s token bill by 60%, 90%, 98%. I believe most of these numbers are real in the setup where they were measured. What I never know is whether they survive contact with my setup. So I tried to build the stack myself in C , layer by layer, and measure each layer with token counts that come from the runtime, not from a chars / 4 guess. Some savings reproduced. One made things worse in the configuration everyone recommends. One fell apart the moment I changed a detail nobody talks about. Here is what I tried, what I measured, and what I could not test at all. Every number in this post is one of three kinds, and I say which: If a source only reached me through a newsletter, I say that too. I wanted the counts to come from a real tokenizer and a real KV cache, so I used llama-server from llama.cpp, which reports two numbers on every response: That second number is the closest local analogue to “uncached input tokens” on a paid API. Now the confession. The machine I ran this on could not download model weights. So I took the Qwen2 tokenizer that ships inside the llama.cpp repo, wrapped it in a tiny random-weight model with the Qwen2 architecture, and served that with the official Qwen2.5 chat template. The model’s text output is garbage. That’s fine for this experiment, because: What this means in practice: my numbers tell you what a prompt layout costs . They tell you nothing about whether a real model would pick the right tool with that layout. I’ll come back to that, because it’s the biggest hole in this post. The session is eight turns of an on-call scenario: check crashing pod logs, find the Jira ticket, look at slow queries, read a README, check CI runs, grep for timeouts, post to Slack, find a meeting slot. Two of the tool outputs are real files from the llama.cpp checkout its README and a real grep -rn -i timeout over the server sources . The rest are synthetic but sized like what those servers return. The pod log is 600 lines of noise with one real crash in the middle. The tool catalog is 60 MCP-style tools across eight servers GitHub, Jira, Postgres, filesystem, Slack, Kubernetes, calendar, Confluence . For the 120-tool test I duplicated them as legacy tools, which is ugly but honestly not far from what I've seen in real orgs that run two versions of the same MCP server. Before any session logic, I sent the same one-word prompt hi with different tool lists and subtracted the bare prompt. TOOLS IN LIST FULL LIST TOKENS tool search + call tool TOKENS CHANGE---------------------------------------------------------------------------------15 1,952 278 -85.8%30 3,399 278 -91.8%60 5,975 278 -95.3%120 12,617 278 -97.8% Measured by me, Qwen2 tokenizer, Qwen2.5 chat template. This is the number that makes tool search look like a no-brainer. Roughly 100 tokens per tool definition, and two meta tools cost a flat 278. I first read about this in Nithin’s article on Toolboxes and Tool Search in Microsoft Foundry https://www.c-sharpcorner.com/article/cut-agent-token-costs-with-toolboxes-and-tool-search-in-microsoft-foundry . The Microsoft Learn page https://learn.microsoft.com/en-my/Azure/foundry/agents/how-to/tools/tool-search confirms the mechanics: The headline numbers, as Nithin reports them from Microsoft’s own benchmark, are over 60% fewer input tokens at 50 tools and over 97% at 1,000. Two things matter about that claim. It’s Microsoft’s benchmark, not something Nithin or I measured. And the baseline was prompt-cached. Hold on to that second detail. Here is my local version. First, plain BM25: // Bm25.csusing System.Text.RegularExpressions;namespace TokenStack;// Plain Okapi BM25. Same family Foundry says it uses for toolbox search.public sealed class Bm25<T { private readonly List< T Item, Dictionary<string, int Tf, int Len docs = new ; private readonly Dictionary<string, int df = new ; private readonly double k1, b; private double avgLen; public Bm25 IEnumerable< T Item, string Text docs, double k1 = 1.2, double b = 0.75 { k1 = k1; b = b; foreach var item, text in docs { var terms = Tokenize text ; var tf = terms.GroupBy t = t .ToDictionary g = g.Key, g = g.Count ; foreach var term in tf.Keys df term = df.GetValueOrDefault term + 1; docs.Add item, tf, terms.Count ; } avgLen = docs.Count == 0 ? 1 : docs.Average d = d.Len ; } public IReadOnlyList< T Item, double Score Search string query, int limit { var q = Tokenize query .Distinct .ToList ; int n = docs.Count; return docs .Select d = { double s = 0; foreach var term in q { if d.Tf.TryGetValue term, out var f continue; var df = df term ; var idf = Math.Log 1 + n - df + 0.5 / df + 0.5 ; s += idf f k1 + 1 / f + k1 1 - b + b d.Len / avgLen ; } return d.Item, Score: s ; } .Where x = x.Score 0 .OrderByDescending x = x.Score .Take limit .ToList ; } private static readonly HashSet<string Stop = new "a an the of to in on for and or is are be it this that with by from as at me my show get".Split ' ' ; // Splits snake case, camelCase and punctuation, lowercases, drops stop words, // and strips a trailing "s" so "logs" matches "log". Crude, but deterministic. public static List<string Tokenize string text { var spaced = Regex.Replace text, " a-z0-9 A-Z ", "$1 $2" ; return Regex.Split spaced.ToLowerInvariant , " ^a-z0-9 +" .Where t = t.Length 1 && Stop.Contains t .Select t = t.Length 3 && t.EndsWith 's' && t.EndsWith "ss" ? t ..^1 : t .ToList ; }} Then the meta tool layer. It indexes whatever your MCP servers return from tools/list, exposes the two meta tools plus anything pinned, and renders search hits as full schemas: // ToolSearch.csusing System.Text.Json;using System.Text.Json.Nodes;namespace TokenStack;public sealed record McpTool string Name, string Description, JsonElement InputSchema, bool Pin = false, string? ExtraSearchText = null ;// A local copy of the toolbox search idea: the model sees two meta tools// plus anything pinned instead of the whole catalog.public sealed class ToolSearch{ public const int DefaultLimit = 5, MaxLimit = 10; private readonly Dictionary<string, McpTool byName; private readonly Bm25<McpTool index; public ToolSearch IEnumerable<McpTool tools { byName = tools.ToDictionary t = t.Name ; index = new Bm25<McpTool byName.Values.Select t = t, SearchText t ; } // Name, description, parameter names and descriptions, plus optional extra keywords. private static string SearchText McpTool t { var parts = new List<string { t.Name, t.Description }; if t.InputSchema.TryGetProperty "properties", out var props foreach var p in props.EnumerateObject { parts.Add p.Name ; if p.Value.TryGetProperty "description", out var d parts.Add d.GetString ?? "" ; } if t.ExtraSearchText is not null parts.Add t.ExtraSearchText ; return string.Join ' ', parts ; } public IReadOnlyList<McpTool Search string query, int limit = DefaultLimit = index.Search query, Math.Clamp limit, 1, MaxLimit .Select r = r.Item .ToList ; public bool TryGet string name, out McpTool tool = byName.TryGetValue name, out tool ; // What the model gets back from tool search: full schemas of the hits only. public static string RenderResults IEnumerable<McpTool hits = JsonSerializer.Serialize hits.Select h = new { name = h.Name, description = h.Description, inputSchema = h.InputSchema } ; // The tool list sent to the model when search is on. public IEnumerable<JsonObject MetaToolDefinitions { yield return Fn "tool search", "Find tools for a capability. Describe what you need in plain words. Returns up to 5 tool schemas max 10 .", """{"type":"object","properties":{"query":{"type":"string","description":"What you need to do"},"limit":{"type":"integer","description":"1 to 10, default 5"}},"required": "query" }""" ; yield return Fn "call tool", "Call a tool returned by tool search, by exact name, with arguments matching its inputSchema.", """{"type":"object","properties":{"name":{"type":"string"},"arguments":{"type":"object"}},"required": "name","arguments" }""" ; foreach var t in byName.Values.Where t = t.Pin yield return ToFunction t ; } public static JsonObject ToFunction McpTool t = Fn t.Name, t.Description, t.InputSchema.GetRawText ; private static JsonObject Fn string name, string description, string schemaJson = new { "type" = "function", "function" = new JsonObject { "name" = name, "description" = description, "parameters" = JsonNode.Parse schemaJson } }; public static List<McpTool Load string path = JsonDocument.Parse File.ReadAllText path .RootElement.EnumerateArray .Select e = new McpTool e.GetProperty "name" .GetString , e.GetProperty "description" .GetString , e.GetProperty "inputSchema" .Clone .ToList ;} tools.json is just an array of MCP tool definitions, the same shape tools/list returns: { "name": "kubernetes get pod logs", "description": "Fetch container logs from a pod, optionally the previous crashed container.", "inputSchema": { "type": "object", "properties": { "namespace": { "type": "string", "description": "Namespace" }, "pod": { "type": "string", "description": "Pod name" }, "previous": { "type": "boolean", "description": "Logs of previous instance" } }, "required": "namespace", "pod" } } This part needs no model at all, so it’s the cleanest measurement in the post. I wrote 32 short queries the way I’d phrase them “restart the checkout deployment”, “are there blocking locks right now” and labeled the right tool for each. CATALOG QUERIES RIGHT TOOL AT 1 RIGHT TOOL IN TOP 5----------------------------------------------------------------60 tools 32 27 31120 tools 32 27 30 Measured by me. The obvious bias: I wrote the queries and the tool descriptions, so the vocabulary overlaps more than it would between a stranger’s model and a stranger’s MCP server. The misses were instructive, though. “find all csproj files” ranked edit file above search files, because "file" matches everywhere and nothing in my query said "search" or "glob". That's exactly what additional search text is for. Now the real test: the eight-turn session, run three ways. All three with the prompt cache warm. Here is 60 tools, with the output gate from the next section turned on I’ll explain why the gate had to be on in a moment : 60 TOOLS, GATED, CACHE WARM REQUESTS TOTAL PROMPT UNCACHED prompt n PEAK PROMPT-------------------------------------------------------------------------------------------full tool list 16 212,279 11,315 17,339tool search 24 212,792 14,326 14,653dynamic top-5 list 16 125,261 67,600 11,855 Measured by me. I stared at the middle row for a while. Tool search at 60 tools saved nothing on total prompt tokens +0.2% and increased uncached tokens by 26.6%. Why? Two reasons, and both are obvious in hindsight. First, the prompt cache already ate most of the tool list’s cost. The full tool list sits at the start of the prompt. It gets processed once, then every later request reads it from cache. In uncached terms, a fixed 6K-token tool list is close to a one-time fee. Second, the extra round trip isn’t free. With search, every turn makes three requests instead of two, and each request carries the whole conversation so far. By turn five my conversation was 10K tokens, so the extra tool search request re-sent all of that just to save a few thousand tokens of tool definitions. Here's the per-request log for the first three turns: REQUEST PROMPT TOKENS PROCESSED FROM CACHE-----------------------------------------------------tool search 328 1 327call tool 730 402 328answer 4,923 4,193 730tool search 4,965 42 4,923call tool 5,344 379 4,965answer 5,943 599 5,344tool search 5,980 37 5,943call tool 6,182 202 5,980answer 6,752 570 6,182 Measured by me. The cache works beautifully here: each tool search call processes only a few dozen new tokens. But the total keeps climbing, and if your provider bills cache reads at all, those totals are what you pay for. Now the scaling picture, which is where tool search earns its keep: GATED, CACHE WARM SEARCH vs FULL: TOTAL SEARCH vs FULL: UNCACHED DYNAMIC vs FULL: UNCACHED-------------------------------------------------------------------------------------------------15 tools +39.5% +20.8% +486.8%60 tools +0.2% +26.6% +497.4%120 tools -31.5% +30.7% +504.6% Derived from my measurements. So in my session, the break-even for tool search on total prompt tokens was around 60 tools, not 10 to 15. At 120 tools it saved 31.5% of total prompt tokens and cut the peak prompt by 37%, which matters for context limits. But uncached tokens went up at every scale, because the extra round trip always adds new tokens to process. I don’t think this contradicts Microsoft. Their benchmark compared request sizes against a cached baseline. My session has big tool outputs and a long history, so the cost of an extra round trip is high. A session with short histories and small outputs would favor search earlier. The lesson I took: the threshold depends on how heavy your conversation is, not just on how many tools you have. Measure your own session shape before you trust anyone’s tool count. The dynamic approach looked best on total tokens -41% at 60 tools . Then I looked at the uncached column: 67,600 versus 11,315, almost 6x worse. The reason: the tool list renders near the top of the prompt. Change it, and every byte after it is a cache miss. Swapping the list every turn means re-processing the whole conversation every turn. On top of that, BM25 on the raw user message missed the needed tool in 1 of 8 turns at 60 tools and 2 of 8 at 120. In a real run, those turns fail. Foundry’s design gets this right, and I only appreciated it after measuring the alternative. The tool list stays fixed tool search, call tool, pinned tools , and the discovered schemas come back as tool results , appended at the end. The prefix never changes, so the cache keeps working. This layer turned out to matter more than tool search. My first attempt at the session ran with a 32K context and no output gate. It died at turn six, in every configuration. The peak prompt without a gate was 48,751 tokens at 60 tools. The pod logs alone were 27,550 tokens. With the default 32K context of a typical Qwen2.5 deployment, the agent physically could not finish an eight-turn on-call session. I had to restart the server with a 128K context just to get the ungated numbers. Two people pushed me toward gating. The first was the Spotify Portal write-up by Dimitri Mazmanov on the Spotify Engineering blog https://engineering.atspotify.com/2026/9/portal-by-spotify-cut-my-claude-code-token-usage-by-90 . It uses a PreToolUse hook that blocks reads of files over 350 lines SHUNT MIN LINES and sends them to a cheaper worker model. The reported result is about 90% savings on the bulk-read work. I also saw Wasowski's breakdown of his Claude Code bill pointing at the same file-size gate, but I only read it through a weekly digest and couldn't find it again to link, so treat that as secondhand. The second was Context Mode https://github.com/mksglu/claude-context-mode , an MIT-licensed MCP server that keeps raw tool output out of the context. It runs commands in a sandbox, indexes output into SQLite FTS5 with BM25, and lets the model search it. The project reports a 98% cut, 315 KB of raw output becoming 5.4 KB over a session. That’s the project’s own benchmark, and I didn’t run it. My gate is much cruder. Under a token budget, output passes through. Over it, the full output goes to disk, and the model gets a header plus the three chunks that best match the user’s request: // OutputGate.csusing System.Text.RegularExpressions;namespace TokenStack;// Tool-output size gate. Small outputs pass through. Big ones go to disk,// and the model gets a header plus the chunks that best match the intent.// alwaysKeep: chunks matching this regex are kept first, whatever BM25 says.public sealed class OutputGate Func<string, Task<int countTokens, int maxTokens = 1000, int chunkLines = 30, int topChunks = 3, string? alwaysKeep = null, string dir = "tool-output" { public async Task< string Text, int RawTokens, int SentTokens ApplyAsync string callId, string intent, string output { int raw = await countTokens output ; if raw <= maxTokens return output, raw, raw ; Directory.CreateDirectory dir ; var path = Path.Combine dir, $"{callId}.txt" ; await File.WriteAllTextAsync path, output ; var lines = output.Split '\n' ; var chunks = Enumerable.Range 0, lines.Length + chunkLines - 1 / chunkLines .Select i = Start: i chunkLines, Text: string.Join '\n', lines.Skip i chunkLines .Take chunkLines .ToList ; var keep = alwaysKeep is null ? new List< int Start, string Text : chunks.Where c = Regex.IsMatch c.Text, alwaysKeep .Take topChunks .ToList ; var ranked = new Bm25< int Start, string Text chunks.Select c = c, c.Text .Search intent, topChunks .Select h = h.Item ; var best = keep.Concat ranked .Distinct .Take topChunks .OrderBy c = c.Start .ToList ; if best.Count == 0 best = chunks.Take 1 .ToList ; // nothing matched: show the head var body = string.Join "\n...\n", best.Select c = $" lines {c.Start + 1}-{c.Start + chunkLines} \n{c.Text}" ; var text = $" output gated: {raw} tokens, {lines.Length} lines saved to {path}. " + $"Showing {best.Count} best-matching chunks for: \"{intent}\". Ask for a line range if you need more. \n{body}"; return text, raw, await countTokens text ; }} Across the session, the eight tool outputs added up to 41,993 tokens raw and 10,580 gated, a 74.8% cut measured . That’s nowhere near Context Mode’s reported 98%, and I’m not surprised: they index everything and let the model query it, while I just pick three chunks and hope. For each turn I had marked the line that actually answers the question. Seven of eight survived the gate. The one that didn’t was the most important line in the whole session: the crash. The user asked, “Why is the payments-api pod crashing in prod?” The answer was a NullReferenceException in PaymentRetryPolicy.cs at line 48. BM25 scored chunks by overlap with the user's words. But "crashing", "pod" and "container" don't appear in a stack trace, and "Payments.Api" appears on every single line. So the crash chunk lost to noise. The model would have been told, confidently, that the logs looked fine. The fix was dull: always keep chunks that look like errors, then fill the rest with BM25. POD LOGS 27,550 TOKENS RAW SENT TOKENS CRASH LINE KEPT?------------------------------------------------------------------gate v1: BM25 on intent only 4,117 nogate v2: + always keep errors 4,072 yes Measured by me. On the other seven outputs, v2 kept every answer line too, and the only size change was the grep output growing by 110 tokens, because the error rule pulled in a different chunk. The session numbers in this post use v1, because I found the bug after the session runs. The token totals barely move between versions, but the honest statement is that the gated session I measured would have misdiagnosed turn one. I keep coming back to this. A 75% cut that silently drops the root cause is worse than no cut. If you gate tool output, check what survives , not just how much you saved. Spotify’s post makes a related point: their cheap worker missed a subtle thread-safety bug that the frontier model caught in seconds. On every request where I ran with cache prompt: false, the processed tokens exactly equaled the prompt tokens. So cold cost is simply the total prompt column. Here's what the cache does, 60 tools, full list: 60 TOOLS, FULL LIST TOTAL PROMPT UNCACHED, COLD UNCACHED, WARM--------------------------------------------------------------------------------no gate 620,853 620,853 42,727with gate v1 212,279 212,279 11,315 Measured by me. A warm cache cut processed tokens by 93.1% without touching anything else. I saw about 2K tokens of run-to-run wobble in the warm numbers, depending on what was left in the server’s cache from the previous configuration. Treat the warm column as plus or minus 2K. That one row made me reorder my priorities: don’t break the prefix. Anything that changes early in the prompt tool lists, system prompt timestamps, reordered tools, a mode switch throws away the biggest saving in this whole post. If you’d rather use Ollama, the client below works against its OpenAI-compatible endpoint. Two caveats: Ollama has no /tokenize endpoint, so the gate needs another token counter, and it doesn't return llama.cpp's timings block. I didn't run these numbers on Ollama, so check how its usage fields behave on a cache hit before you compare them with mine. This is the piece I’d copy into any agent project, even without the rest. It logs what the runtime says, per request: // RuntimeClient.csusing System.Net.Http.Json;using System.Text.Json;using System.Text.Json.Nodes;namespace TokenStack;// Per-request numbers as the runtime reports them, not estimates.// PromptTokens: whole prompt. Evaluated: tokens actually processed cache misses .public sealed record TurnUsage string Label, int PromptTokens, int Evaluated, int Cached ;// Talks to any OpenAI-compatible endpoint. Tested against llama-server.// llama-server adds "timings" prompt n, cache n and honours "cache prompt".// Ollama's /v1 endpoint returns usage.prompt tokens; see the article for its caveat.public sealed class RuntimeClient string baseUrl, string model { private readonly HttpClient http = new { BaseAddress = new Uri baseUrl , Timeout = TimeSpan.FromMinutes 10 }; public async Task<TurnUsage SendAsync string label, JsonArray messages, JsonArray? tools, bool cachePrompt { var body = new JsonObject { "model" = model, "messages" = messages.DeepClone , "max tokens" = 1, // we only want the prompt-side counters "temperature" = 0, "cache prompt" = cachePrompt }; if tools is { Count: 0 } body "tools" = tools.DeepClone ; using var res = await http.PostAsJsonAsync "/v1/chat/completions", body ; var json = await res.Content.ReadFromJsonAsync<JsonElement ; if res.IsSuccessStatusCode throw new InvalidOperationException json.ToString ; int prompt = json.GetProperty "usage" .GetProperty "prompt tokens" .GetInt32 ; int evaluated = prompt, cached = 0; if json.TryGetProperty "timings", out var t { evaluated = t.GetProperty "prompt n" .GetInt32 ; cached = t.TryGetProperty "cache n", out var c ? c.GetInt32 : prompt - evaluated; } return new TurnUsage label, prompt, evaluated, cached ; } // llama-server only. Exact tokenizer count for a string. public async Task<int CountTokensAsync string text { using var res = await http.PostAsJsonAsync "/tokenize", new { content = text } ; var json = await res.Content.ReadFromJsonAsync<JsonElement ; return json.GetProperty "tokens" .GetArrayLength ; }} The harness that runs all four experiments is Program.cs. It expects tools.json, tools-120.json, and a fixtures/ folder with turns.json the scripted turns , queries.json the labeled recall queries and the tool output files. // Program.csusing System.Text.Json;using System.Text.Json.Nodes;using TokenStack;// Usage:// dotnet run -- tooltax http://localhost:8080// dotnet run -- recall// dotnet run -- gate http://localhost:8080// dotnet run -- session http://localhost:8080 60var mode = args.ElementAtOrDefault 0 ?? "session";var url = args.ElementAtOrDefault 1 ?? "http://localhost:8080";var model = Environment.GetEnvironmentVariable "MODEL" ?? "local";var all60 = ToolSearch.Load "tools.json" ;var all120 = ToolSearch.Load "tools-120.json" ;var turns = JsonDocument.Parse File.ReadAllText "fixtures/turns.json" .RootElement.EnumerateArray .ToList ;var targets = turns.Select t = t.GetProperty "tool" .GetString .ToHashSet ;// Tool sets of 15/30/60/120 that always contain the 8 tools the session needs.List<McpTool Subset int n = n switch{ 120 = all120, 60 = all60, = all60.Where t = targets.Contains t.Name .Concat all60.Where t = targets.Contains t.Name .Take n - targets.Count .ToList };switch mode { case "tooltax": await ToolTax ; break; case "recall": Recall ; break; case "gate": await Gate ; break; default: await Session int.Parse args.ElementAtOrDefault 2 ?? "60" ; break;}// 1. What does the tool list alone cost? Same one-line prompt, different tool lists.async Task ToolTax { var rt = new RuntimeClient url, model ; var msgs = new JsonArray { Msg "user", "hi" }; int bare = await rt.SendAsync "bare", msgs, null, false .PromptTokens; foreach var n in new { 15, 30, 60, 120 } { var tools = new JsonArray Subset n .Select t = JsonNode ToolSearch.ToFunction t .ToArray ; var full = await rt.SendAsync $"full{n}", msgs, tools, false .PromptTokens - bare; var meta = new JsonArray new ToolSearch Subset n .MetaToolDefinitions .Select x = JsonNode x .ToArray ; var search = await rt.SendAsync $"meta{n}", msgs, meta, false .PromptTokens - bare; Console.WriteLine $"tooltax,{n},{full},{search}" ; }}// 2. Does BM25 put the right tool in the top 5? Labeled queries, no model involved.void Recall { var labeled = JsonDocument.Parse File.ReadAllText "fixtures/queries.json" .RootElement.EnumerateArray .Select e = Q: e.GetProperty "query" .GetString , Tool: e.GetProperty "tool" .GetString .Concat turns.Select t = Q: t.GetProperty "q1" .GetString , Tool: t.GetProperty "tool" .GetString .ToList ; foreach var name, set in new { "60", all60 , "120", all120 } { var ts = new ToolSearch set ; int at1 = 0, at5 = 0; foreach var q, tool in labeled { var hits = ts.Search q, 5 .Select h = h.Name .ToList ; if hits.FirstOrDefault == tool at1++; if hits.Contains tool at5++; else Console.WriteLine $"miss,{name},\"{q}\",{tool},top1={hits.FirstOrDefault }" ; } Console.WriteLine $"recall,{name},{labeled.Count},{at1},{at5}" ; }}// 2b. Gate v1 BM25 on intent only vs v2 also always keep error-looking chunks , per tool output.async Task Gate { var rt = new RuntimeClient url, model ; var v1 = new OutputGate rt.CountTokensAsync ; var v2 = new OutputGate rt.CountTokensAsync, alwaysKeep: @" ?i \b fatal|error|exception|failure|failed \b" ; foreach var t in turns { var raw = File.ReadAllText Path.Combine "fixtures", t.GetProperty "output" .GetString ; string user = t.GetProperty "user" .GetString , needle = t.GetProperty "needle" .GetString ; var a = await v1.ApplyAsync "v1", user, raw ; var b = await v2.ApplyAsync "v2", user, raw ; Console.WriteLine $"gate,{t.GetProperty "tool" .GetString },{a.RawTokens},{a.SentTokens},{a.Text.Contains needle },{b.SentTokens},{b.Text.Contains needle }" ; }}// 3. The scripted 8-turn session. Tool choices are scripted so every configuration does// identical work; only the prompt shape changes. Counters come from the runtime.async Task Session int n { var rt = new RuntimeClient url, model ; var tools = Subset n ; var search = new ToolSearch tools ; var fullList = new JsonArray tools.Select t = JsonNode ToolSearch.ToFunction t .ToArray ; var metaList = new JsonArray search.MetaToolDefinitions .Select x = JsonNode x .ToArray ; foreach var config in new { "full", "search", "dynamic" } foreach var gated in new { false, true } { // Cold runs cache prompt=false evaluated exactly PromptTokens on every request I ran, // so cold cost = the PromptTokens column. Set COLD=1 to measure it again yourself. var warm = Environment.GetEnvironmentVariable "COLD" = "1"; var gate = new OutputGate rt.CountTokensAsync ; var log = new List<TurnUsage ; int misses = 0, extraSearches = 0, needlesLost = 0, toolOutRaw = 0, toolOutSent = 0, callId = 0; var msgs = new JsonArray { Msg "system", "You are an on-call assistant for the payments team. Use tools when needed. Be brief." }; foreach var t in turns { string user = t.GetProperty "user" .GetString , tool = t.GetProperty "tool" .GetString ; var toolArgs = JsonNode.Parse t.GetProperty "args" .GetRawText ; msgs.Add Msg "user", user ; JsonArray list = config switch { "full" = fullList, "search" = metaList, = new JsonArray search.Search user .Select x = JsonNode ToolSearch.ToFunction x .ToArray }; if config == "dynamic" && search.Search user .Any x = x.Name == tool misses++; async Task Step string label = log.Add await rt.SendAsync label, msgs, list, warm ; if config == "search" { foreach var q in new { t.GetProperty "q1" .GetString , t.GetProperty "q2" .GetString } { await Step "tool search" ; var hits = search.Search q ; AddCall msgs, ++callId, "tool search", new JsonObject { "query" = q }, ToolSearch.RenderResults hits ; if hits.Any h = h.Name == tool break; extraSearches++; } await Step "call tool" ; } else await Step "tool call" ; var raw = File.ReadAllText Path.Combine "fixtures", t.GetProperty "output" .GetString ; var sent = raw; if gated { var r = await gate.ApplyAsync $"call{callId + 1}", user, raw ; sent = r.Text; toolOutRaw += r.RawTokens; toolOutSent += r.SentTokens; if sent.Contains t.GetProperty "needle" .GetString { needlesLost++; if warm Console.WriteLine $"needle lost,{n},{config},{tool},raw={r.RawTokens}" ; } } var name, a = config == "search" ? "call tool", JsonNode new JsonObject { "name" = tool, "arguments" = toolArgs } : tool, toolArgs ; AddCall msgs, ++callId, name, a, sent ; await Step "answer" ; msgs.Add Msg "assistant", t.GetProperty "answer" .GetString ; } Console.WriteLine string.Join ',', "session", n, config, gated ? "gate" : "nogate", warm ? "warm" : "cold", log.Count, log.Sum x = x.PromptTokens , log.Sum x = x.Evaluated , log.Max x = x.PromptTokens , misses, extraSearches, needlesLost, toolOutRaw, toolOutSent ; if n == 60 && warm && config == "search" foreach var u in log Console.WriteLine $"turnlog,{ gated ? "gate" : "nogate" },{u.Label},{u.PromptTokens},{u.Evaluated},{u.Cached}" ; }}static JsonObject Msg string role, string content = new { "role" = role, "content" = content };static void AddCall JsonArray msgs, int id, string name, JsonNode args, string result { msgs.Add new JsonObject { "role" = "assistant", "content" = "", "tool calls" = new JsonArray new JsonObject { "id" = $"call {id}", "type" = "function", "function" = new JsonObject { "name" = name, "arguments" = args.ToJsonString } } } ; msgs.Add new JsonObject { "role" = "tool", "tool call id" = $"call {id}", "content" = result } ;} One scripted turn in fixtures/turns.json looks like this. q1 and q2 are the queries the scripted model sends to tool search, and needle is the line that must survive the gate: { "user": "Why is the payments-api pod crashing in prod? Check the logs of the previous container.", "tool": "kubernetes get pod logs", "args": { "namespace": "prod", "pod": "payments-api-7c9f", "previous": true }, "q1": "pod container logs", "q2": "kubernetes get pod logs previous container", "output": "pod logs.txt", "needle": "PaymentRetryPolicy.cs:line 48", "answer": "A NullReferenceException in PaymentRetryPolicy.GetDelay at line 48 kills the process." } With llama.cpp, using a real Qwen2.5 model and the same template I used: Build llama.cpp CPU git clone --depth 1 https://github.com/ggml-org/llama.cppcd llama.cppcmake -B buildcmake --build build -j --target llama-server Serve Qwen2.5-7B-Instruct with tool-calling templates and a single slot./build/bin/llama-server \ -hf Qwen/Qwen2.5-7B-Instruct-GGUF:Q4 K M \ --jinja \ --chat-template-file models/templates/Qwen-Qwen2.5-7B-Instruct.jinja \ -c 32768 -np 1 --port 8080 Or with Docker: docker run --rm -p 8080:8080 -v llama-cache:/root/.cache \ ghcr.io/ggml-org/llama.cpp:server \ -hf Qwen/Qwen2.5-7B-Instruct-GGUF:Q4 K M --jinja -c 32768 -np 1 \ --host 0.0.0.0 --port 8080 Then the harness: dotnet new console -n TokenStackcd TokenStack copy Bm25.cs, ToolSearch.cs, OutputGate.cs, RuntimeClient.cs and Program.cs here, plus tools.json, tools-120.json and the fixtures/ folderdotnet run -c Release -- tooltax http://localhost:8080dotnet run -c Release -- recalldotnet run -c Release -- gate http://localhost:8080dotnet run -c Release -- session http://localhost:8080 60 Keep -c 32768 if you want to watch the ungated session hit the wall like mine did. Use -c 131072 if you want the ungated numbers. For Ollama, the session and tool-list modes work against its OpenAI-compatible endpoint: ollama pull qwen2.5:7bMODEL=qwen2.5:7b dotnet run -c Release -- session http://localhost:11434 60 Since Ollama has no /tokenize, swap the gate's counter for a llama-server tokenizer or an estimate: // Estimate only. Good enough to trigger the gate, not good enough to report savings.var gate = new OutputGate text = Task.FromResult text.Length / 4 ; Several things on my list I could not test from my setup. I’m including them because they shaped my thinking, but none of their numbers are mine. Claude Fast Mode. Per Anthropic’s fast mode docs https://platform.claude.com/docs/en/build-with-claude/fast-mode , which Mikhail Petrusheuski walks through in his .NET routing piece https://medium.com/@mikhail.petrusheuski/claude-fast-mode-changes-a-net-agents-routing-policy-not-its-intelligence-de17b46b908c : you set speed: "fast" on the request; Opus 5.5 gets up to 2.5x output tokens per second at $8 input and $40 output per million tokens; the response's usage.speed tells you which speed actually ran; and switching between fast and standard is a prompt-cache miss, because the two speeds don't share cached prefixes. Fast mode doesn't cut tokens, it buys speed. But that cache detail connects straight to my data. In my session, the gated prompt was about 10.5K tokens by turn five, and the ungated one was about 38K. Flip speeds mid-session, and all of that becomes uncached again derived from my runs, Qwen tokenizer, so Claude's counts will differ . I agree with Mikhail's framing: treat speed as a per-session routing decision, log requested versus actual speed, and don't toggle it per request. The turn-inspector mod. The Code newsletter October 13 pointed me to a Claude Code mod you install with npx claude-code-templates@latest --mod ui/turn-inspector and open with /turns to see per-turn usage. Its listing page https://www.aitmpl.com/component/mods/ui/turn-inspector confirms the install command but doesn't document what each turn shows. I'd avoid running an unpinned @latest for something that hooks into your agent's lifecycle. Pin the version you actually reviewed: npm view claude-code-templates versionnpx claude-code-templates@"$ npm view claude-code-templates version " --mod ui/turn-inspector That still installs today’s latest, but at least you know which version, and you can write it down. Reflection’s Beam. Reflection reports about 25,000 tokens per attempt on Terminal-Bench 2.1 at roughly 80%, versus about 80,000 tokens for GLM-5.2 at a similar score. Those are vendor numbers, which I read in an AI Weekly summary https://aiweekly.co/alerts/reflection-ai-open-weights-beam-a-501b-apache-20-model-hitting-80-on-terminal and a newsletter. It’s an open-weights 501B model, and I didn’t have the hardware to try it. The idea is the interesting part for me: model choice is a token lever too, and a model that takes fewer, better steps beats any prompt trimming I did here. Atlassian Teamwork Graph. Atlassian’s announcement of Code Context in the Teamwork Graph https://seekingalpha.com/pr/20617213-atlassian-announces-code-context-bringing-codebase-understanding-into-the-teamwork-graph claims coding agents were 44% more accurate with 48% fewer tokens. That’s an internal benchmark, which I only saw summarized in The Code October 15 . Same pattern as the gate: give the agent pre-structured context instead of raw dumps. Foundry Agent Optimizer. Nithin also wrote a guide to the Agent Optimizer https://www.c-sharpcorner.com/article/let-foundry-improve-your-agent-a-practical-guide-to-the-agent-optimizer , and one line in it stuck with me: the optimizer targets quality, not tokens, and better instructions are often longer. Worth remembering when a tool promises to “improve” your agent. LAYER MEASURED ON TOKEN CHANGE STATUS------------------------------------------------------------------------------------------------------------------------Tool list - tool search/call tool single request, 60 tools -95.3% tool-list tokens measured single request, 120 tools -97.8% tool-list tokens measuredPrompt cache warm session, 60 tools, no gate -93.1% uncached measuredOutput gate v1 session, 60 tools -65.8% total, -73.5% uncached, measured lost crash line in 1 of 8 turnsOutput gate v2 keep errors pod logs only -85.2% vs raw, crash line kept measuredTool search on top of gate session, 15 tools +39.5% total, +20.8% uncached measured session, 60 tools +0.2% total, +26.6% uncached measured session, 120 tools -31.5% total, +30.7% uncached measuredDynamic top-5 tool list session, 60 tools -41.0% total, +497.4% uncached, measured needed tool missing 1 of 8 turnsFoundry Tool Search Microsoft benchmark 60% at 50 tools, 97% at 1,000 reported, not reproducedContext Mode project benchmark -98% raw tool output reported, not reproducedSpotify file-size gate Spotify, Java monorepo ~90% on bulk-read work reported, not reproducedReflection Beam Terminal-Bench 2.1 ~25K vs ~80K tokens per attempt vendor, not reproducedAtlassian Teamwork Graph internal benchmark +44% accuracy, -48% tokens vendor, not reproducedClaude Fast Mode Anthropic docs no token change, cache miss on docs, not tested switch, up to 2.5x output speed The biggest gap: I never measured whether a real model picks the right tool through tool search as reliably as it does from a full list. My BM25 recall was 31 of 32 on queries I wrote myself, and a model writing its own queries against someone else's tool descriptions will do worse. If search costs one extra failed call per session, the totals above shift. I haven't tested that, and I'd want to before putting tool search in production. I also don’t know how my numbers translate to a paid API bill. Cached tokens are billed at a discount on most providers, so the “total prompt” and “uncached” columns are both part of the cost, weighted by your provider’s pricing. And Claude’s tokenizer is not Qwen’s, so absolute counts will differ even if the ratios hold. Next, I want to run the same scripted session against a real model with tool choice unscripted, and count failed searches. If you’ve done that already, I’d genuinely like to see your numbers. Tags: dotnet, csharp, ai-agents, llm, mcp, llamacpp, ollama, prompt-caching, token-optimization, cost-optimization Cutting Agent Token Costs in .NET: Tool Search, Output Gating and What the Numbers Really Say https://pub.towardsai.net/cutting-agent-token-costs-in-net-tool-search-output-gating-and-what-the-numbers-really-say-d9e3f85ff30f was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.