# Cutting Agent Token Costs in .NET: Tool Search, Output Gating and What the Numbers Really Say > Source: > Published: 2026-10-26 19:01:02+00:00 Every few days another post tells me I can cut my agent’s token bill by 60%, 90%, 98%. I believe most of these numbers are real in the setup where they were measured. What I never know is whether they survive contact with my setup. So I tried to build the stack myself in C#, layer by layer, and measure each layer with token counts that come from the runtime, not from a chars / 4 guess. Some savings reproduced. One made things worse in the configuration everyone recommends. One fell apart the moment I changed a detail nobody talks about. Here is what I tried, what I measured, and what I could not test at all. Every number in this post is one of three kinds, and I say which: If a source only reached me through a newsletter, I say that too. I wanted the counts to come from a real tokenizer and a real KV cache, so I used llama-server from llama.cpp, which reports two numbers on every response: That second number is the closest local analogue to “uncached input tokens” on a paid API. Now the confession. The machine I ran this on could not download model weights. So I took the Qwen2 tokenizer that ships inside the llama.cpp repo, wrapped it in a tiny random-weight model with the Qwen2 architecture, and served that with the official Qwen2.5 chat template. The model’s text output is garbage. That’s fine for this experiment, because: What this means in practice: my numbers tell you what a prompt layout **costs**. They tell you nothing about whether a real model would pick the right tool with that layout. I’ll come back to that, because it’s the biggest hole in this post. The session is eight turns of an on-call scenario: check crashing pod logs, find the Jira ticket, look at slow queries, read a README, check CI runs, grep for timeouts, post to Slack, find a meeting slot. Two of the tool outputs are real files from the llama.cpp checkout (its README and a real grep -rn -i timeout over the server sources). The rest are synthetic but sized like what those servers return. The pod log is 600 lines of noise with one real crash in the middle. The tool catalog is 60 MCP-style tools across eight servers (GitHub, Jira, Postgres, filesystem, Slack, Kubernetes, calendar, Confluence). For the 120-tool test I duplicated them as legacy_* tools, which is ugly but honestly not far from what I've seen in real orgs that run two versions of the same MCP server. Before any session logic, I sent the same one-word prompt (hi) with different tool lists and subtracted the bare prompt. ``` TOOLS IN LIST FULL LIST TOKENS tool_search + call_tool TOKENS CHANGE---------------------------------------------------------------------------------15 1,952 278 -85.8%30 3,399 278 -91.8%60 5,975 278 -95.3%120 12,617 278 -97.8% ``` Measured by me, Qwen2 tokenizer, Qwen2.5 chat template. This is the number that makes tool search look like a no-brainer. Roughly 100 tokens per tool definition, and two meta tools cost a flat 278. I first read about this in Nithin’s article on [Toolboxes and Tool Search in Microsoft Foundry](https://www.c-sharpcorner.com/article/cut-agent-token-costs-with-toolboxes-and-tool-search-in-microsoft-foundry). The [Microsoft Learn page](https://learn.microsoft.com/en-my/Azure/foundry/agents/how-to/tools/tool-search) confirms the mechanics: The headline numbers, as Nithin reports them from Microsoft’s own benchmark, are over 60% fewer input tokens at 50 tools and over 97% at 1,000. Two things matter about that claim. It’s Microsoft’s benchmark, not something Nithin or I measured. And the baseline was prompt-cached. Hold on to that second detail. Here is my local version. First, plain BM25: ``` // Bm25.csusing System.Text.RegularExpressions;namespace TokenStack;// Plain Okapi BM25. Same family Foundry says it uses for toolbox_search.public sealed class Bm25{ private readonly List<(T Item, Dictionary Tf, int Len)> _docs = new(); private readonly Dictionary _df = new(); private readonly double _k1, _b; private double _avgLen; public Bm25(IEnumerable<(T Item, string Text)> docs, double k1 = 1.2, double b = 0.75) { _k1 = k1; _b = b; foreach (var (item, text) in docs) { var terms = Tokenize(text); var tf = terms.GroupBy(t => t).ToDictionary(g => g.Key, g => g.Count()); foreach (var term in tf.Keys) _df[term] = _df.GetValueOrDefault(term) + 1; _docs.Add((item, tf, terms.Count)); } _avgLen = _docs.Count == 0 ? 1 : _docs.Average(d => d.Len); } public IReadOnlyList<(T Item, double Score)> Search(string query, int limit) { var q = Tokenize(query).Distinct().ToList(); int n = _docs.Count; return _docs .Select(d => { double s = 0; foreach (var term in q) { if (!d.Tf.TryGetValue(term, out var f)) continue; var df = _df[term]; var idf = Math.Log(1 + (n - df + 0.5) / (df + 0.5)); s += idf * f * (_k1 + 1) / (f + _k1 * (1 - _b + _b * d.Len / _avgLen)); } return (d.Item, Score: s); }) .Where(x => x.Score > 0) .OrderByDescending(x => x.Score) .Take(limit) .ToList(); } private static readonly HashSet Stop = new( "a an the of to in on for and or is are be it this that with by from as at me my show get".Split(' ')); // Splits snake_case, camelCase and punctuation, lowercases, drops stop words, // and strips a trailing "s" so "logs" matches "log". Crude, but deterministic. public static List Tokenize(string text) { var spaced = Regex.Replace(text, "([a-z0-9])([A-Z])", "$1 $2"); return Regex.Split(spaced.ToLowerInvariant(), "[^a-z0-9]+") .Where(t => t.Length > 1 && !Stop.Contains(t)) .Select(t => t.Length > 3 && t.EndsWith('s') && !t.EndsWith("ss") ? t[..^1] : t) .ToList(); }} ``` Then the meta tool layer. It indexes whatever your MCP servers return from tools/list, exposes the two meta tools plus anything pinned, and renders search hits as full schemas: ``` // ToolSearch.csusing System.Text.Json;using System.Text.Json.Nodes;namespace TokenStack;public sealed record McpTool(string Name, string Description, JsonElement InputSchema, bool Pin = false, string? ExtraSearchText = null);// A local copy of the toolbox_search idea: the model sees two meta tools// (plus anything pinned) instead of the whole catalog.public sealed class ToolSearch{ public const int DefaultLimit = 5, MaxLimit = 10; private readonly Dictionary _byName; private readonly Bm25 _index; public ToolSearch(IEnumerable tools) { _byName = tools.ToDictionary(t => t.Name); _index = new Bm25(_byName.Values.Select(t => (t, SearchText(t)))); } // Name, description, parameter names and descriptions, plus optional extra keywords. private static string SearchText(McpTool t) { var parts = new List { t.Name, t.Description }; if (t.InputSchema.TryGetProperty("properties", out var props)) foreach (var p in props.EnumerateObject()) { parts.Add(p.Name); if (p.Value.TryGetProperty("description", out var d)) parts.Add(d.GetString() ?? ""); } if (t.ExtraSearchText is not null) parts.Add(t.ExtraSearchText); return string.Join(' ', parts); } public IReadOnlyList Search(string query, int limit = DefaultLimit) => _index.Search(query, Math.Clamp(limit, 1, MaxLimit)).Select(r => r.Item).ToList(); public bool TryGet(string name, out McpTool tool) => _byName.TryGetValue(name, out tool!); // What the model gets back from tool_search: full schemas of the hits only. public static string RenderResults(IEnumerable hits) => JsonSerializer.Serialize(hits.Select(h => new { name = h.Name, description = h.Description, inputSchema = h.InputSchema })); // The tool list sent to the model when search is on. public IEnumerable MetaToolDefinitions() { yield return Fn("tool_search", "Find tools for a capability. Describe what you need in plain words. Returns up to 5 tool schemas (max 10).", """{"type":"object","properties":{"query":{"type":"string","description":"What you need to do"},"limit":{"type":"integer","description":"1 to 10, default 5"}},"required":["query"]}"""); yield return Fn("call_tool", "Call a tool returned by tool_search, by exact name, with arguments matching its inputSchema.", """{"type":"object","properties":{"name":{"type":"string"},"arguments":{"type":"object"}},"required":["name","arguments"]}"""); foreach (var t in _byName.Values.Where(t => t.Pin)) yield return ToFunction(t); } public static JsonObject ToFunction(McpTool t) => Fn(t.Name, t.Description, t.InputSchema.GetRawText()); private static JsonObject Fn(string name, string description, string schemaJson) => new() { ["type"] = "function", ["function"] = new JsonObject { ["name"] = name, ["description"] = description, ["parameters"] = JsonNode.Parse(schemaJson) } }; public static List Load(string path) => JsonDocument.Parse(File.ReadAllText(path)).RootElement.EnumerateArray() .Select(e => new McpTool(e.GetProperty("name").GetString()!, e.GetProperty("description").GetString()!, e.GetProperty("inputSchema").Clone())) .ToList();} ``` tools.json is just an array of MCP tool definitions, the same shape tools/list returns: ``` [ { "name": "kubernetes__get_pod_logs", "description": "Fetch container logs from a pod, optionally the previous crashed container.", "inputSchema": { "type": "object", "properties": { "namespace": { "type": "string", "description": "Namespace" }, "pod": { "type": "string", "description": "Pod name" }, "previous": { "type": "boolean", "description": "Logs of previous instance" } }, "required": ["namespace", "pod"] } }] ``` This part needs no model at all, so it’s the cleanest measurement in the post. I wrote 32 short queries the way I’d phrase them (“restart the checkout deployment”, “are there blocking locks right now”) and labeled the right tool for each. ``` CATALOG QUERIES RIGHT TOOL AT #1 RIGHT TOOL IN TOP 5----------------------------------------------------------------60 tools 32 27 31120 tools 32 27 30 ``` Measured by me. The obvious bias: I wrote the queries and the tool descriptions, so the vocabulary overlaps more than it would between a stranger’s model and a stranger’s MCP server. The misses were instructive, though. “find all csproj files” ranked edit_file above search_files, because "file" matches everywhere and nothing in my query said "search" or "glob". That's exactly what additional_search_text is for. Now the real test: the eight-turn session, run three ways. All three with the prompt cache warm. Here is 60 tools, with the output gate from the next section turned on (I’ll explain why the gate had to be on in a moment): ``` 60 TOOLS, GATED, CACHE WARM REQUESTS TOTAL PROMPT UNCACHED (prompt_n) PEAK PROMPT-------------------------------------------------------------------------------------------full tool list 16 212,279 11,315 17,339tool search 24 212,792 14,326 14,653dynamic top-5 list 16 125,261 67,600 11,855 ``` Measured by me. I stared at the middle row for a while. Tool search at 60 tools saved **nothing** on total prompt tokens (+0.2%) and **increased** uncached tokens by 26.6%. Why? Two reasons, and both are obvious in hindsight. **First, the prompt cache already ate most of the tool list’s cost.** The full tool list sits at the start of the prompt. It gets processed once, then every later request reads it from cache. In uncached terms, a fixed 6K-token tool list is close to a one-time fee. **Second, the extra round trip isn’t free.** With search, every turn makes three requests instead of two, and each request carries the whole conversation so far. By turn five my conversation was 10K tokens, so the extra tool_search request re-sent all of that just to save a few thousand tokens of tool definitions. Here's the per-request log for the first three turns: ``` REQUEST PROMPT TOKENS PROCESSED FROM CACHE-----------------------------------------------------tool_search 328 1 327call_tool 730 402 328answer 4,923 4,193 730tool_search 4,965 42 4,923call_tool 5,344 379 4,965answer 5,943 599 5,344tool_search 5,980 37 5,943call_tool 6,182 202 5,980answer 6,752 570 6,182 ``` Measured by me. The cache works beautifully here: each tool_search call processes only a few dozen new tokens. But the total keeps climbing, and if your provider bills cache reads at all, those totals are what you pay for. Now the scaling picture, which is where tool search earns its keep: ``` GATED, CACHE WARM SEARCH vs FULL: TOTAL SEARCH vs FULL: UNCACHED DYNAMIC vs FULL: UNCACHED-------------------------------------------------------------------------------------------------15 tools +39.5% +20.8% +486.8%60 tools +0.2% +26.6% +497.4%120 tools -31.5% +30.7% +504.6% ``` Derived from my measurements. So in my session, the break-even for tool search on total prompt tokens was around 60 tools, not 10 to 15. At 120 tools it saved 31.5% of total prompt tokens and cut the peak prompt by 37%, which matters for context limits. But uncached tokens went up at every scale, because the extra round trip always adds new tokens to process. I don’t think this contradicts Microsoft. Their benchmark compared request sizes against a cached baseline. My session has big tool outputs and a long history, so the cost of an extra round trip is high. A session with short histories and small outputs would favor search earlier. The lesson I took: **the threshold depends on how heavy your conversation is, not just on how many tools you have.** Measure your own session shape before you trust anyone’s tool count. The dynamic approach looked best on total tokens (-41% at 60 tools). Then I looked at the uncached column: **67,600 versus 11,315, almost 6x worse.** The reason: the tool list renders near the top of the prompt. Change it, and every byte after it is a cache miss. Swapping the list every turn means re-processing the whole conversation every turn. On top of that, BM25 on the raw user message missed the needed tool in 1 of 8 turns at 60 tools and 2 of 8 at 120. In a real run, those turns fail. Foundry’s design gets this right, and I only appreciated it after measuring the alternative. The tool list stays fixed (tool_search, call_tool, pinned tools), and the discovered schemas come back as **tool results**, appended at the end. The prefix never changes, so the cache keeps working. This layer turned out to matter more than tool search. My first attempt at the session ran with a 32K context and no output gate. It died at turn six, in every configuration. The peak prompt without a gate was 48,751 tokens at 60 tools. The pod logs alone were 27,550 tokens. With the default 32K context of a typical Qwen2.5 deployment, the agent physically could not finish an eight-turn on-call session. I had to restart the server with a 128K context just to get the ungated numbers. Two people pushed me toward gating. The first was the Spotify Portal write-up by Dimitri Mazmanov on the [Spotify Engineering blog](https://engineering.atspotify.com/2026/9/portal-by-spotify-cut-my-claude-code-token-usage-by-90). It uses a PreToolUse hook that blocks reads of files over 350 lines (SHUNT_MIN_LINES) and sends them to a cheaper worker model. The reported result is about 90% savings on the bulk-read work. I also saw Wasowski's breakdown of his Claude Code bill pointing at the same file-size gate, but I only read it through a weekly digest and couldn't find it again to link, so treat that as secondhand. The second was [Context Mode](https://github.com/mksglu/claude-context-mode), an MIT-licensed MCP server that keeps raw tool output out of the context. It runs commands in a sandbox, indexes output into SQLite FTS5 with BM25, and lets the model search it. The project reports a 98% cut, 315 KB of raw output becoming 5.4 KB over a session. That’s the project’s own benchmark, and I didn’t run it. My gate is much cruder. Under a token budget, output passes through. Over it, the full output goes to disk, and the model gets a header plus the three chunks that best match the user’s request: ``` // OutputGate.csusing System.Text.RegularExpressions;namespace TokenStack;// Tool-output size gate. Small outputs pass through. Big ones go to disk,// and the model gets a header plus the chunks that best match the intent.// alwaysKeep: chunks matching this regex are kept first, whatever BM25 says.public sealed class OutputGate(Func> countTokens, int maxTokens = 1000, int chunkLines = 30, int topChunks = 3, string? alwaysKeep = null, string dir = "tool-output"){ public async Task<(string Text, int RawTokens, int SentTokens)> ApplyAsync(string callId, string intent, string output) { int raw = await countTokens(output); if (raw <= maxTokens) return (output, raw, raw); Directory.CreateDirectory(dir); var path = Path.Combine(dir, $"{callId}.txt"); await File.WriteAllTextAsync(path, output); var lines = output.Split('\n'); var chunks = Enumerable.Range(0, (lines.Length + chunkLines - 1) / chunkLines) .Select(i => (Start: i * chunkLines, Text: string.Join('\n', lines.Skip(i * chunkLines).Take(chunkLines)))) .ToList(); var keep = alwaysKeep is null ? new List<(int Start, string Text)>() : chunks.Where(c => Regex.IsMatch(c.Text, alwaysKeep)).Take(topChunks).ToList(); var ranked = new Bm25<(int Start, string Text)>(chunks.Select(c => (c, c.Text))) .Search(intent, topChunks) .Select(h => h.Item); var best = keep.Concat(ranked).Distinct().Take(topChunks).OrderBy(c => c.Start).ToList(); if (best.Count == 0) best = chunks.Take(1).ToList(); // nothing matched: show the head var body = string.Join("\n...\n", best.Select(c => $"[lines {c.Start + 1}-{c.Start + chunkLines}]\n{c.Text}")); var text = $"[output gated: {raw} tokens, {lines.Length} lines saved to {path}. " + $"Showing {best.Count} best-matching chunks for: \"{intent}\". Ask for a line range if you need more.]\n{body}"; return (text, raw, await countTokens(text)); }} ``` Across the session, the eight tool outputs added up to 41,993 tokens raw and 10,580 gated, a **74.8% cut** (measured). That’s nowhere near Context Mode’s reported 98%, and I’m not surprised: they index everything and let the model query it, while I just pick three chunks and hope. For each turn I had marked the line that actually answers the question. Seven of eight survived the gate. The one that didn’t was the most important line in the whole session: the crash. The user asked, “Why is the payments-api pod crashing in prod?” The answer was a NullReferenceException in PaymentRetryPolicy.cs at line 48. BM25 scored chunks by overlap with the user's words. But "crashing", "pod" and "container" don't appear in a stack trace, and "Payments.Api" appears on every single line. So the crash chunk lost to noise. The model would have been told, confidently, that the logs looked fine. The fix was dull: always keep chunks that look like errors, then fill the rest with BM25. ``` POD LOGS (27,550 TOKENS RAW) SENT TOKENS CRASH LINE KEPT?------------------------------------------------------------------gate v1: BM25 on intent only 4,117 nogate v2: + always keep errors 4,072 yes ``` Measured by me. On the other seven outputs, v2 kept every answer line too, and the only size change was the grep output growing by 110 tokens, because the error rule pulled in a different chunk. The session numbers in this post use v1, because I found the bug after the session runs. The token totals barely move between versions, but the honest statement is that the gated session I measured would have misdiagnosed turn one. I keep coming back to this. A 75% cut that silently drops the root cause is worse than no cut. If you gate tool output, **check what survives**, not just how much you saved. Spotify’s post makes a related point: their cheap worker missed a subtle thread-safety bug that the frontier model caught in seconds. On every request where I ran with cache_prompt: false, the processed tokens exactly equaled the prompt tokens. So cold cost is simply the total prompt column. Here's what the cache does, 60 tools, full list: ``` 60 TOOLS, FULL LIST TOTAL PROMPT UNCACHED, COLD UNCACHED, WARM--------------------------------------------------------------------------------no gate 620,853 620,853 42,727with gate (v1) 212,279 212,279 11,315 ``` Measured by me. A warm cache cut processed tokens by 93.1% without touching anything else. I saw about 2K tokens of run-to-run wobble in the warm numbers, depending on what was left in the server’s cache from the previous configuration. Treat the warm column as plus or minus 2K. That one row made me reorder my priorities: **don’t break the prefix.** Anything that changes early in the prompt (tool lists, system prompt timestamps, reordered tools, a mode switch) throws away the biggest saving in this whole post. If you’d rather use Ollama, the client below works against its OpenAI-compatible endpoint. Two caveats: Ollama has no /tokenize endpoint, so the gate needs another token counter, and it doesn't return llama.cpp's timings block. I didn't run these numbers on Ollama, so check how its usage fields behave on a cache hit before you compare them with mine. This is the piece I’d copy into any agent project, even without the rest. It logs what the runtime says, per request: ``` // RuntimeClient.csusing System.Net.Http.Json;using System.Text.Json;using System.Text.Json.Nodes;namespace TokenStack;// Per-request numbers as the runtime reports them, not estimates.// PromptTokens: whole prompt. Evaluated: tokens actually processed (cache misses).public sealed record TurnUsage(string Label, int PromptTokens, int Evaluated, int Cached);// Talks to any OpenAI-compatible endpoint. Tested against llama-server.// llama-server adds "timings" (prompt_n, cache_n) and honours "cache_prompt".// Ollama's /v1 endpoint returns usage.prompt_tokens; see the article for its caveat.public sealed class RuntimeClient(string baseUrl, string model){ private readonly HttpClient _http = new() { BaseAddress = new Uri(baseUrl), Timeout = TimeSpan.FromMinutes(10) }; public async Task SendAsync(string label, JsonArray messages, JsonArray? tools, bool cachePrompt) { var body = new JsonObject { ["model"] = model, ["messages"] = messages.DeepClone(), ["max_tokens"] = 1, // we only want the prompt-side counters ["temperature"] = 0, ["cache_prompt"] = cachePrompt }; if (tools is { Count: > 0 }) body["tools"] = tools.DeepClone(); using var res = await _http.PostAsJsonAsync("/v1/chat/completions", body); var json = await res.Content.ReadFromJsonAsync(); if (!res.IsSuccessStatusCode) throw new InvalidOperationException(json.ToString()); int prompt = json.GetProperty("usage").GetProperty("prompt_tokens").GetInt32(); int evaluated = prompt, cached = 0; if (json.TryGetProperty("timings", out var t)) { evaluated = t.GetProperty("prompt_n").GetInt32(); cached = t.TryGetProperty("cache_n", out var c) ? c.GetInt32() : prompt - evaluated; } return new TurnUsage(label, prompt, evaluated, cached); } // llama-server only. Exact tokenizer count for a string. public async Task CountTokensAsync(string text) { using var res = await _http.PostAsJsonAsync("/tokenize", new { content = text }); var json = await res.Content.ReadFromJsonAsync(); return json.GetProperty("tokens").GetArrayLength(); }} ``` The harness that runs all four experiments is Program.cs. It expects tools.json, tools-120.json, and a fixtures/ folder with turns.json (the scripted turns), queries.json (the labeled recall queries) and the tool output files. ``` // Program.csusing System.Text.Json;using System.Text.Json.Nodes;using TokenStack;// Usage:// dotnet run -- tooltax http://localhost:8080// dotnet run -- recall// dotnet run -- gate http://localhost:8080// dotnet run -- session http://localhost:8080 60var mode = args.ElementAtOrDefault(0) ?? "session";var url = args.ElementAtOrDefault(1) ?? "http://localhost:8080";var model = Environment.GetEnvironmentVariable("MODEL") ?? "local";var all60 = ToolSearch.Load("tools.json");var all120 = ToolSearch.Load("tools-120.json");var turns = JsonDocument.Parse(File.ReadAllText("fixtures/turns.json")).RootElement.EnumerateArray().ToList();var targets = turns.Select(t => t.GetProperty("tool").GetString()!).ToHashSet();// Tool sets of 15/30/60/120 that always contain the 8 tools the session needs.List Subset(int n) => n switch{ 120 => all120, 60 => all60, _ => all60.Where(t => targets.Contains(t.Name)).Concat(all60.Where(t => !targets.Contains(t.Name)).Take(n - targets.Count)).ToList()};switch (mode){ case "tooltax": await ToolTax(); break; case "recall": Recall(); break; case "gate": await Gate(); break; default: await Session(int.Parse(args.ElementAtOrDefault(2) ?? "60")); break;}// 1. What does the tool list alone cost? Same one-line prompt, different tool lists.async Task ToolTax(){ var rt = new RuntimeClient(url, model); var msgs = new JsonArray { Msg("user", "hi") }; int bare = (await rt.SendAsync("bare", msgs, null, false)).PromptTokens; foreach (var n in new[] { 15, 30, 60, 120 }) { var tools = new JsonArray(Subset(n).Select(t => (JsonNode)ToolSearch.ToFunction(t)).ToArray()); var full = (await rt.SendAsync($"full{n}", msgs, tools, false)).PromptTokens - bare; var meta = new JsonArray(new ToolSearch(Subset(n)).MetaToolDefinitions().Select(x => (JsonNode)x).ToArray()); var search = (await rt.SendAsync($"meta{n}", msgs, meta, false)).PromptTokens - bare; Console.WriteLine($"tooltax,{n},{full},{search}"); }}// 2. Does BM25 put the right tool in the top 5? Labeled queries, no model involved.void Recall(){ var labeled = JsonDocument.Parse(File.ReadAllText("fixtures/queries.json")).RootElement.EnumerateArray() .Select(e => (Q: e.GetProperty("query").GetString()!, Tool: e.GetProperty("tool").GetString()!)) .Concat(turns.Select(t => (Q: t.GetProperty("q1").GetString()!, Tool: t.GetProperty("tool").GetString()!))) .ToList(); foreach (var (name, set) in new[] { ("60", all60), ("120", all120) }) { var ts = new ToolSearch(set); int at1 = 0, at5 = 0; foreach (var (q, tool) in labeled) { var hits = ts.Search(q, 5).Select(h => h.Name).ToList(); if (hits.FirstOrDefault() == tool) at1++; if (hits.Contains(tool)) at5++; else Console.WriteLine($"miss,{name},\"{q}\",{tool},top1={hits.FirstOrDefault()}"); } Console.WriteLine($"recall,{name},{labeled.Count},{at1},{at5}"); }}// 2b. Gate v1 (BM25 on intent only) vs v2 (also always keep error-looking chunks), per tool output.async Task Gate(){ var rt = new RuntimeClient(url, model); var v1 = new OutputGate(rt.CountTokensAsync); var v2 = new OutputGate(rt.CountTokensAsync, alwaysKeep: @"(?i)\b(fatal|error|exception|failure|failed)\b"); foreach (var t in turns) { var raw = File.ReadAllText(Path.Combine("fixtures", t.GetProperty("output").GetString()!)); string user = t.GetProperty("user").GetString()!, needle = t.GetProperty("needle").GetString()!; var a = await v1.ApplyAsync("v1", user, raw); var b = await v2.ApplyAsync("v2", user, raw); Console.WriteLine($"gate,{t.GetProperty("tool").GetString()},{a.RawTokens},{a.SentTokens},{a.Text.Contains(needle)},{b.SentTokens},{b.Text.Contains(needle)}"); }}// 3. The scripted 8-turn session. Tool choices are scripted so every configuration does// identical work; only the prompt shape changes. Counters come from the runtime.async Task Session(int n){ var rt = new RuntimeClient(url, model); var tools = Subset(n); var search = new ToolSearch(tools); var fullList = new JsonArray(tools.Select(t => (JsonNode)ToolSearch.ToFunction(t)).ToArray()); var metaList = new JsonArray(search.MetaToolDefinitions().Select(x => (JsonNode)x).ToArray()); foreach (var config in new[] { "full", "search", "dynamic" }) foreach (var gated in new[] { false, true }) { // Cold runs (cache_prompt=false) evaluated exactly PromptTokens on every request I ran, // so cold cost = the PromptTokens column. Set COLD=1 to measure it again yourself. var warm = Environment.GetEnvironmentVariable("COLD") != "1"; var gate = new OutputGate(rt.CountTokensAsync); var log = new List(); int misses = 0, extraSearches = 0, needlesLost = 0, toolOutRaw = 0, toolOutSent = 0, callId = 0; var msgs = new JsonArray { Msg("system", "You are an on-call assistant for the payments team. Use tools when needed. Be brief.") }; foreach (var t in turns) { string user = t.GetProperty("user").GetString()!, tool = t.GetProperty("tool").GetString()!; var toolArgs = JsonNode.Parse(t.GetProperty("args").GetRawText())!; msgs.Add(Msg("user", user)); JsonArray list = config switch { "full" => fullList, "search" => metaList, _ => new JsonArray(search.Search(user).Select(x => (JsonNode)ToolSearch.ToFunction(x)).ToArray()) }; if (config == "dynamic" && !search.Search(user).Any(x => x.Name == tool)) misses++; async Task Step(string label) => log.Add(await rt.SendAsync(label, msgs, list, warm)); if (config == "search") { foreach (var q in new[] { t.GetProperty("q1").GetString()!, t.GetProperty("q2").GetString()! }) { await Step("tool_search"); var hits = search.Search(q); AddCall(msgs, ++callId, "tool_search", new JsonObject { ["query"] = q }, ToolSearch.RenderResults(hits)); if (hits.Any(h => h.Name == tool)) break; extraSearches++; } await Step("call_tool"); } else await Step("tool_call"); var raw = File.ReadAllText(Path.Combine("fixtures", t.GetProperty("output").GetString()!)); var sent = raw; if (gated) { var r = await gate.ApplyAsync($"call{callId + 1}", user, raw); sent = r.Text; toolOutRaw += r.RawTokens; toolOutSent += r.SentTokens; if (!sent.Contains(t.GetProperty("needle").GetString()!)) { needlesLost++; if (warm) Console.WriteLine($"needle_lost,{n},{config},{tool},raw={r.RawTokens}"); } } var (name, a) = config == "search" ? ("call_tool", (JsonNode)new JsonObject { ["name"] = tool, ["arguments"] = toolArgs }) : (tool, toolArgs); AddCall(msgs, ++callId, name, a, sent); await Step("answer"); msgs.Add(Msg("assistant", t.GetProperty("answer").GetString()!)); } Console.WriteLine(string.Join(',', "session", n, config, gated ? "gate" : "nogate", warm ? "warm" : "cold", log.Count, log.Sum(x => x.PromptTokens), log.Sum(x => x.Evaluated), log.Max(x => x.PromptTokens), misses, extraSearches, needlesLost, toolOutRaw, toolOutSent)); if (n == 60 && warm && config == "search") foreach (var u in log) Console.WriteLine($"turnlog,{(gated ? "gate" : "nogate")},{u.Label},{u.PromptTokens},{u.Evaluated},{u.Cached}"); }}static JsonObject Msg(string role, string content) => new() { ["role"] = role, ["content"] = content };static void AddCall(JsonArray msgs, int id, string name, JsonNode args, string result){ msgs.Add(new JsonObject { ["role"] = "assistant", ["content"] = "", ["tool_calls"] = new JsonArray(new JsonObject { ["id"] = $"call_{id}", ["type"] = "function", ["function"] = new JsonObject { ["name"] = name, ["arguments"] = args.ToJsonString() } }) }); msgs.Add(new JsonObject { ["role"] = "tool", ["tool_call_id"] = $"call_{id}", ["content"] = result });} ``` One scripted turn in fixtures/turns.json looks like this. q1 and q2 are the queries the scripted model sends to tool_search, and needle is the line that must survive the gate: ``` [ { "user": "Why is the payments-api pod crashing in prod? Check the logs of the previous container.", "tool": "kubernetes__get_pod_logs", "args": { "namespace": "prod", "pod": "payments-api-7c9f", "previous": true }, "q1": "pod container logs", "q2": "kubernetes get pod logs previous container", "output": "pod_logs.txt", "needle": "PaymentRetryPolicy.cs:line 48", "answer": "A NullReferenceException in PaymentRetryPolicy.GetDelay at line 48 kills the process." }] ``` With llama.cpp, using a real Qwen2.5 model and the same template I used: ``` # Build llama.cpp (CPU)git clone --depth 1 https://github.com/ggml-org/llama.cppcd llama.cppcmake -B buildcmake --build build -j --target llama-server# Serve Qwen2.5-7B-Instruct with tool-calling templates and a single slot./build/bin/llama-server \ -hf Qwen/Qwen2.5-7B-Instruct-GGUF:Q4_K_M \ --jinja \ --chat-template-file models/templates/Qwen-Qwen2.5-7B-Instruct.jinja \ -c 32768 -np 1 --port 8080 ``` Or with Docker: ``` docker run --rm -p 8080:8080 -v llama-cache:/root/.cache \ ghcr.io/ggml-org/llama.cpp:server \ -hf Qwen/Qwen2.5-7B-Instruct-GGUF:Q4_K_M --jinja -c 32768 -np 1 \ --host 0.0.0.0 --port 8080 ``` Then the harness: ``` dotnet new console -n TokenStackcd TokenStack# copy Bm25.cs, ToolSearch.cs, OutputGate.cs, RuntimeClient.cs and Program.cs here,# plus tools.json, tools-120.json and the fixtures/ folderdotnet run -c Release -- tooltax http://localhost:8080dotnet run -c Release -- recalldotnet run -c Release -- gate http://localhost:8080dotnet run -c Release -- session http://localhost:8080 60 ``` Keep -c 32768 if you want to watch the ungated session hit the wall like mine did. Use -c 131072 if you want the ungated numbers. For Ollama, the session and tool-list modes work against its OpenAI-compatible endpoint: ``` ollama pull qwen2.5:7bMODEL=qwen2.5:7b dotnet run -c Release -- session http://localhost:11434 60 ``` Since Ollama has no /tokenize, swap the gate's counter for a llama-server tokenizer or an estimate: ``` // Estimate only. Good enough to trigger the gate, not good enough to report savings.var gate = new OutputGate(text => Task.FromResult(text.Length / 4)); ``` Several things on my list I could not test from my setup. I’m including them because they shaped my thinking, but none of their numbers are mine. **Claude Fast Mode.** Per [Anthropic’s fast mode docs](https://platform.claude.com/docs/en/build-with-claude/fast-mode), which Mikhail Petrusheuski walks through in [his .NET routing piece](https://medium.com/@mikhail.petrusheuski/claude-fast-mode-changes-a-net-agents-routing-policy-not-its-intelligence-de17b46b908c): you set speed: "fast" on the request; Opus 5.5 gets up to 2.5x output tokens per second at $8 input and $40 output per million tokens; the response's usage.speed tells you which speed actually ran; and switching between fast and standard is a prompt-cache miss, because the two speeds don't share cached prefixes. Fast mode doesn't cut tokens, it buys speed. But that cache detail connects straight to my data. In my session, the gated prompt was about 10.5K tokens by turn five, and the ungated one was about 38K. Flip speeds mid-session, and all of that becomes uncached again (derived from my runs, Qwen tokenizer, so Claude's counts will differ). I agree with Mikhail's framing: treat speed as a per-session routing decision, log requested versus actual speed, and don't toggle it per request. **The turn-inspector mod.** The Code newsletter (October 13) pointed me to a Claude Code mod you install with npx claude-code-templates@latest --mod ui/turn-inspector and open with /turns to see per-turn usage. Its [listing page](https://www.aitmpl.com/component/mods/ui/turn-inspector) confirms the install command but doesn't document what each turn shows. I'd avoid running an unpinned @latest for something that hooks into your agent's lifecycle. Pin the version you actually reviewed: ``` npm view claude-code-templates versionnpx claude-code-templates@"$(npm view claude-code-templates version)" --mod ui/turn-inspector ``` That still installs today’s latest, but at least you know which version, and you can write it down. **Reflection’s Beam.** Reflection reports about 25,000 tokens per attempt on Terminal-Bench 2.1 at roughly 80%, versus about 80,000 tokens for GLM-5.2 at a similar score. Those are vendor numbers, which I read in an [AI Weekly summary](https://aiweekly.co/alerts/reflection-ai-open-weights-beam-a-501b-apache-20-model-hitting-80-on-terminal) and a newsletter. It’s an open-weights 501B model, and I didn’t have the hardware to try it. The idea is the interesting part for me: model choice is a token lever too, and a model that takes fewer, better steps beats any prompt trimming I did here. **Atlassian Teamwork Graph.** Atlassian’s announcement of [Code Context in the Teamwork Graph](https://seekingalpha.com/pr/20617213-atlassian-announces-code-context-bringing-codebase-understanding-into-the-teamwork-graph) claims coding agents were 44% more accurate with 48% fewer tokens. That’s an internal benchmark, which I only saw summarized in The Code (October 15). Same pattern as the gate: give the agent pre-structured context instead of raw dumps. **Foundry Agent Optimizer.** Nithin also wrote a [guide to the Agent Optimizer](https://www.c-sharpcorner.com/article/let-foundry-improve-your-agent-a-practical-guide-to-the-agent-optimizer), and one line in it stuck with me: the optimizer targets quality, not tokens, and better instructions are often longer. Worth remembering when a tool promises to “improve” your agent. ``` LAYER MEASURED ON TOKEN CHANGE STATUS------------------------------------------------------------------------------------------------------------------------Tool list -> tool_search/call_tool single request, 60 tools -95.3% tool-list tokens measured single request, 120 tools -97.8% tool-list tokens measuredPrompt cache warm session, 60 tools, no gate -93.1% uncached measuredOutput gate v1 session, 60 tools -65.8% total, -73.5% uncached, measured lost crash line in 1 of 8 turnsOutput gate v2 (keep errors) pod logs only -85.2% vs raw, crash line kept measuredTool search (on top of gate) session, 15 tools +39.5% total, +20.8% uncached measured session, 60 tools +0.2% total, +26.6% uncached measured session, 120 tools -31.5% total, +30.7% uncached measuredDynamic top-5 tool list session, 60 tools -41.0% total, +497.4% uncached, measured needed tool missing 1 of 8 turnsFoundry Tool Search Microsoft benchmark >60% at 50 tools, >97% at 1,000 reported, not reproducedContext Mode project benchmark -98% raw tool output reported, not reproducedSpotify file-size gate Spotify, Java monorepo ~90% on bulk-read work reported, not reproducedReflection Beam Terminal-Bench 2.1 ~25K vs ~80K tokens per attempt vendor, not reproducedAtlassian Teamwork Graph internal benchmark +44% accuracy, -48% tokens vendor, not reproducedClaude Fast Mode Anthropic docs no token change, cache miss on docs, not tested switch, up to 2.5x output speed ``` The biggest gap: I never measured whether a real model picks the right tool through tool_search as reliably as it does from a full list. My BM25 recall was 31 of 32 on queries I wrote myself, and a model writing its own queries against someone else's tool descriptions will do worse. If search costs one extra failed call per session, the totals above shift. I haven't tested that, and I'd want to before putting tool search in production. I also don’t know how my numbers translate to a paid API bill. Cached tokens are billed at a discount on most providers, so the “total prompt” and “uncached” columns are both part of the cost, weighted by your provider’s pricing. And Claude’s tokenizer is not Qwen’s, so absolute counts will differ even if the ratios hold. Next, I want to run the same scripted session against a real model with tool choice unscripted, and count failed searches. If you’ve done that already, I’d genuinely like to see your numbers. Tags: dotnet, csharp, ai-agents, llm, mcp, llamacpp, ollama, prompt-caching, token-optimization, cost-optimization [Cutting Agent Token Costs in .NET: Tool Search, Output Gating and What the Numbers Really Say](https://pub.towardsai.net/cutting-agent-token-costs-in-net-tool-search-output-gating-and-what-the-numbers-really-say-d9e3f85ff30f) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.