# Route First, Retrieve Second, But Never Trust the Router’s Confidence > Source: > Published: 2026-11-02 22:01:01+00:00 The number that sent me down this hole was a zero. Miodrag Cekikj put a question router in front of three retrieval designs for an invented payments company he calls Naissus Pay. The router was a model with a strict structured output: pick a design, report a confidence, and if the confidence drops below 0.6, fall back to the expensive hybrid. A very normal setup. I have shipped almost exactly that config. Across 150 routing decisions from three different models, the 0.6 fallback fired zero times. Not because the routers were always right. They weren’t. gpt-4.1 reported 0.95 or higher on all 32 of its replayed decisions, including six wrong ones. gpt-4o-mini never went below 0.70 across 50 decisions, and its three costly misses all said 0.90. gpt-5-mini’s lowest confidence was 0.86. The safety net was hung at a height nothing ever fell to. That is Cekikj’s measurement on his own synthetic data, with his own caveats (more on those below). I wanted to know whether it was a quirk of LLM routers or something more general, so I built my own router in .NET, gave it a confidence band, and ran a calibration check on a labelled set to prove or disprove that the confidence was usable. This post is what happened, including the part where my calibration fix made things worse. Every number in this section is Cekikj’s, from his “Route First, Retrieve Second” piece. He replayed strategies over 32 answered questions with recorded gpt-4.1 outcomes and computed ground truth, plus 18 routing-only questions written to test whether the router reads the shape of a question or just its keywords. ``` +----------------------------------+----------+-------------------+| Strategy (Cekikj, 32 questions) | Accuracy | Cost per question |+----------------------------------+----------+-------------------+| Run all three, keep best | 92% | 1.99 cents || Oracle (cheapest correct design) | 92% | 0.58 cents || Always hybrid | 91% | 0.97 cents || Question routers (3 models) | 83% | 0.63 to 0.81 c || Router + empty-graph fallback | 91% | 0.75 cents |+----------------------------------+----------+-------------------+Source: Miodrag Cekikj, author-run replay on synthetic data. ``` Two results from that piece changed how I think about this. The first is the fallback that did work. Instead of trusting confidence, he escalated to hybrid whenever the graph query came back empty or the agent answered without querying. That rule fired on exactly the five trap questions (with gpt-4o-mini routing) and lifted accuracy from 83% to 91%, matching always-hybrid at 0.75 cents instead of 0.97. An observed result beat a reported probability. The second is the data-aware hint. A 242-token description of what the graph actually holds, added to the router prompt, took the routers to 97.5% on his 20 matrix questions once the missing structure had been built. He is upfront that the hint was written knowing which facts the trap questions needed, and that the escalation rule was replayed, not deployed. The sample is one invented company. I’d read 97.5% as “the router was starving for information about the data”, not as a number to expect. In a companion piece, “Not a Retrieval Problem, a Shape Problem”, Cekikj compares classic RAG with graph traversal on questions that are chains, sets, absences and definitions. Classic RAG scored 13.6% on those shapes. Graph traversal scored 100%. I want to be careful with that 100%. The ground truth was generated from the same structure the graph was built from. If the truth is computed by walking a graph, then walking that graph returns the truth. That isn’t a finding about graphs; it’s close to a tautology. What the experiment does show convincingly is the other half: top-k retrieval over passages cannot count, cannot enumerate a set it only partly retrieved, and cannot prove something is missing. That half doesn’t depend on how the truth was generated. His “Written Down or Worked Out” piece frames the same idea as two questions you ask before picking an architecture: is the answer written down or worked out, and does the knowledge live in text or in connected data? The nasty cell is “text, worked out”: the answer has to be assembled from many documents, and no retrieval trick fixes it. On his 20-question set, more passages and whole-corpus prompting both helped a little and cost a lot. Building the structure first is what moved the number. I ran into the tautology problem myself, and I’ll show you exactly where. Routers and gates look like different components. A router decides which path a question takes; a gate decides whether an action may run. Underneath, both are cheap classifiers whose output we treat as a decision, and both fail the same way: they can only rule on what reaches them, and they report confidence about things they never saw. Wasowski’s piece on agent gates and model routers makes this point with AmPermBench, which he describes as an independent stress test of Claude Code’s auto mode. In that benchmark, 36.8% of state changes (93 of 253) bypassed the permission classifier entirely, because they went through edits to files inside the project. During artifact cleanup the miss rate reached 92.9% (78 of 84 dangerous actions). His point that stuck with me: the classifier didn’t get those wrong, because nobody asked it. In his words, “What passes through the gate is still set by the harness.” The underlying paper is arXiv 2604.04978; I read it through Wasowski’s summary and a feed post, not end to end. Two other numbers have been circulating with Wasowski’s name attached: an 81% end-to-end miss rate, and an independent Jev router test of 825 samples at 51% accuracy. I couldn’t find either in his piece or trace them to a primary source, so I’m not leaning on them here. If you know where they come from, I’d like to read it. Part of why routers and gates are suddenly everywhere is that they got cheap. TypeSafe’s Jev and now Microsoft-Decision-1 are models that take a situation, a question and a fixed set of options, and return a probability per option. The pricing invites you to put one in front of every agent step. Before I did that, I sorted the numbers by who produced them. ``` +--------------------------------------+--------------------------+-------------------------+| Claim | Number | Who measured it |+--------------------------------------+--------------------------+-------------------------+| Jev input price | $0.042 / 1M input tokens | TypeSafe (vendor price) || Jev latency | 70 to 500 ms | TypeSafe (vendor) || MS-Decision-1 input price | $0.042 / 1M, output free | Microsoft (vendor) || MS-Decision-1 speed vs GPT-6 Sol | about 35x faster (P50) | Microsoft (vendor) || MS-Decision-1 accuracy | best on 36-bench set | Microsoft, own set || Jev phishing, one question | 62.6% (Haiku 4.5: 81.3%) | Independent repo || Jev phishing, split into five | 95.0% (Haiku 4.5: 93.2%) | Independent repo || Jev answers flipped by one injected | 12 to 22% | Braintrust, via || instruction | | Kalimatov's roundup || Simple Jev threshold router vs best | worse at 9 of 9 | rishirajFS, independent || single-model mix | operating points | |+--------------------------------------+--------------------------+-------------------------+ ``` The independent results came mostly through Akzhan Kalimatov’s roundup of Jev tests. The phishing benchmark used 2,000 synthetic emails, and the five sub-questions were written after studying the dataset, so treat the 95% as an upper bound on what decomposition buys you. The Braintrust injection figure comes with no task, sample size or method in the roundup; I’d check the original before quoting it anywhere serious. The Microsoft-Decision-1 numbers are from Microsoft’s announcement as reported by Gowtham Boyina and others; at the time I looked, I found no third-party results. The pattern I take from the independent column is the same one Cekikj found. These models do well when the question is narrow and the options are described, and badly when one vague question carries all the weight. A third-party writeup I read made it concrete: a router given option names with no descriptions sent all 40 hard tasks to the cheap model at a median confidence of 0.96; one-line descriptions of each option fixed 37 of 40. Confidence was high in both runs. Only the information changed. The rishirajFS benchmark adds a sobering detail I’ll come back to: plain TF-IDF and small embedding routers, with no decision model at all, also beat single-model mixes there. The router doesn’t have to be fancy. It has to be checked. On cost, the NVIDIA researchers’ paper “Small Language Models are the Future of Agentic AI” is often quoted as saying agents overpay 10 to 30x. As Aqeel Abbas points out, it’s a position paper, and the 10 to 30x refers to serving a 7B model versus a 70B to 175B one. It’s an argument about serving cost, not a measured outcome for any agent. I wanted my own numbers, so I built a small replay harness in .NET 8 with no NuGet packages. It has: Now the honest part. The machine I ran this on couldn’t pull Ollama models, so **no LLM is in the measured loop**. The text route’s reader is deterministic: it parses only the passages BM25 retrieved and answers from those. That makes it an upper bound on a model reader’s comprehension, so every text-route failure you see below is a retrieval failure, not a reading failure. The router is a multinomial naive Bayes classifier over word unigrams and bigrams, trained on oracle labels. Naive Bayes is famously overconfident, which makes it a fair stand-in for “a cheap router that reports a probability”, but it is not gpt-4o-mini and I won’t pretend the numbers transfer. The Ollama router and the Ollama escalation rung are in the code and I tested the request and response handling against a stubbed HTTP handler. I didn’t measure them. Where a number would need a model, I’ll say so instead of filling it in. Costs are reported as tokens per question: the characters a reader model would receive, divided by four. Latency is in-process microseconds, which means it measures my retrieval code and nothing else. A real model call would dwarf it. I kept it in the table because the graph route’s 2 microseconds versus BM25’s 430 is still a true statement about the retrieval layer. I split the questions into 800 for training, 300 for dev, 400 for test using phrasings the router had seen, and 300 for a shifted set using phrasings it had never seen. The shifted set also includes one question shape the router never trained on: “if this data store goes down, what breaks, including indirectly?” The router interface is deliberately boring: ``` public enum RouteName { Text, Graph, Hybrid }public record RouteDecision(RouteName Route, double Confidence, IReadOnlyDictionary Probs);public interface IRouter { RouteDecision Decide(string question); } ``` The confidence band is a single threshold fitted on the dev set, never on test: the smallest confidence above which routed decisions were at least 97% right, with a minimum of 20 decisions behind it. Anything below the band goes to hybrid. ``` public static double FitBand(IList<(double Conf, bool Correct)> dev, double target, int minSupport = 20){ foreach (var th in Enumerable.Range(50, 50).Select(i => i / 100.0)) { var above = dev.Where(x => x.Conf >= th).ToList(); if (above.Count >= minSupport && above.Average(x => x.Correct ? 1.0 : 0.0) >= target) return th; } return 1.01; // no usable band: everything goes to hybrid} ``` Note what Correct means here. It's not "the router picked the oracle label". It's "the route the router picked produced the right answer". That distinction cost me an evening, as you'll see. To test whether confidence is usable, I bin decisions by reported confidence and compare the mean confidence in each bin with how often the chosen route was actually right. Expected calibration error (ECE) is the weighted gap across ten equal-width bins. ``` public static double Ece(IList<(double Conf, bool Correct)> xs){ double ece = 0; for (int b = 0; b < 10; b++) { double lo = b / 10.0, hi = (b + 1) / 10.0; var bin = xs.Where(x => x.Conf >= lo && (x.Conf < hi || (b == 9 && x.Conf <= 1.0))).ToList(); if (bin.Count == 0) continue; double conf = bin.Average(x => x.Conf), acc = bin.Average(x => x.Correct ? 1.0 : 0.0); ece += bin.Count * Math.Abs(conf - acc); } return ece / xs.Count;} ``` I expected naive Bayes to fall over here. It didn’t. On the 400 in-distribution test questions: ``` Raw NB router, TEST (seen phrasings), n=400, ECE 0.039+----------------+-----+-----------+-------------+| confidence bin | n | mean conf | route right |+----------------+-----+-----------+-------------+| 0.00 - 0.60 | 9 | 0.539 | 100.0 % || 0.60 - 0.80 | 27 | 0.690 | 88.9 % || 0.80 - 0.90 | 19 | 0.867 | 89.5 % || 0.90 - 0.99 | 46 | 0.951 | 91.3 % || 0.99 - 1.00 | 299 | 1.000 | 98.7 % |+----------------+-----+-----------+-------------+ ``` That is roughly calibrated. But look at the top row. The 0.6 fallback, the exact rule Cekikj used, fired on 9 of 400 questions, and the router was right on all 9 of them. The fallback spent extra tokens on questions that didn’t need it and caught nothing. ``` Raw NB router, SHIFTED (unseen phrasings), n=300, ECE 0.142+----------------+-----+-----------+-------------+| confidence bin | n | mean conf | route right |+----------------+-----+-----------+-------------+| 0.00 - 0.60 | 3 | 0.599 | 100.0 % || 0.60 - 0.80 | 13 | 0.754 | 100.0 % || 0.80 - 0.90 | 48 | 0.876 | 70.8 % || 0.90 - 0.99 | 52 | 0.927 | 71.2 % || 0.99 - 1.00 | 184 | 1.000 | 89.7 % |+----------------+-----+-----------+-------------+ ``` 184 decisions at a reported confidence of essentially 1.00, about one in ten of them wrong. The 0.80 to 0.99 range was right about 71% of the time. The 0.6 fallback fired 3 times. This is Cekikj’s zero, reproduced with a completely different kind of router. The textbook remedy is temperature scaling: fit one temperature on the dev set to soften overconfident probabilities. I did that, and ECE went from 0.039 to 0.128 on test and from 0.142 to 0.175 on the shifted set. It took me a while to see why. I had fitted the temperature against the oracle labels, meaning “did the router pick the cheapest correct route”. But the router often picks graph for a question where text was cheaper and both were right. That decision is wrong by the label and fine by the outcome. I calibrated the router’s confidence about a question nobody downstream was asking. So I refitted the temperature against what I actually care about, whether the chosen route was right: ``` double bestT = 1, bestNll = double.MaxValue;for (double logT = 0; logT <= 6; logT += 0.05){ router.Temperature = Math.Exp(logT); double nll = dev.Average(q => { var d = router.Decide(q.Text); double p = Math.Clamp(d.Confidence, 1e-9, 1 - 1e-9); return RouteWasRight(q, d.Route) ? -Math.Log(p) : -Math.Log(1 - p); }); if (nll < bestNll) { bestNll = nll; bestT = router.Temperature; }}router.Temperature = bestT; ``` That landed at T = 3.0, gave ECE 0.072 on test (still worse than doing nothing) and 0.145 on the shifted set (no better than doing nothing). A calibration fitted on in-distribution dev data cannot know what out-of-distribution phrasing looks like. Of course it can’t. I knew that in the abstract and still expected a little help. The band built on the recalibrated confidence did “work” on the shifted set: it matched always-hybrid at 90.3%. It did so by firing on 103 of 300 questions and costing exactly as much as always-hybrid. Cekikj saw the same thing: raising the threshold changed nothing until everything went to hybrid. A band that only works when it routes everything to the expensive path isn’t a band. One more thing I only caught by printing the decisions. On the shifted set, the raw router sent 216 questions to hybrid, 84 to text and zero to graph. That looks like a cautious router doing the right thing on unfamiliar input. It wasn’t caution. With Laplace smoothing, every unseen word adds a little evidence for whichever class has the fewest training tokens, and hybrid had only 10 training examples out of 800. Unfamiliar phrasing pushed everything toward hybrid, with confidence near 1.00. If my label distribution had been different, the same mechanism would have pushed unfamiliar questions into the cheapest, wrongest route, just as confidently. I nearly wrote a paragraph praising the router’s humility. I also tried my own version of Cekikj’s hint: before features are extracted, the router is told which words name something the graph actually holds (services, teams, stores, incidents, tiers), and a question with no graph entity gets an explicit marker. On seen phrasings it changed nothing (96.8% either way). On the shifted set its lowest confidence was 0.910, it reported 1.00 on 230 decisions that were right 87.4% of the time, and its ECE was 0.081. Better accuracy and a lower ECE on paper, but with every single decision at 0.91 or above, the confidence couldn’t tell me which answers to doubt. My conclusion for this section: **reported confidence was usable in-distribution and useless exactly where I needed it.** I could prove the first half and disprove the second, on my data, with my router. That’s the half that matters, because production traffic is the shifted set. Cekikj’s empty-graph rule worked because it observed something about the result instead of asking the router how it felt. I built a ladder around four such signals: ``` [Flags]public enum Signal { None = 0, EmptyResult = 1, MissingCitation = 2, DanglingReference = 4, SchemaFailure = 8 }public record RouteResult(RouteName Route, string? Answer, IReadOnlyList Citations, int Tokens, Signal Signals, double Micros){ public bool StructurallyOk => Signals == Signal.None && Answer is not null;} ``` The schema check is the cheapest and dumbest of the four: ``` public static Signal Schema(Question q, string? ans){ if (ans is null) return Signal.None; return q.Intent.Kind switch { Kind.Tier => ans is "1" or "2" or "3" ? Signal.None : Signal.SchemaFailure, Kind.TeamTierCount => int.TryParse(ans, out var n) && n >= 0 ? Signal.None : Signal.SchemaFailure, Kind.Owner or Kind.OnCall or Kind.IncidentService => ans.Length > 0 && !ans.Contains(',') ? Signal.None : Signal.SchemaFailure, Kind.Define => ans.Length > 10 ? Signal.None : Signal.SchemaFailure, _ => Signal.None // lists: an empty list is a legal answer, which is exactly the problem };} ``` The ladder runs the routed choice; on any signal it runs the other route; if that also signals, it escalates to a stronger model: ``` (string? Ans, int Tokens, double Micros, bool PastHybrid) Ladder(Question q, RouteName first){ if (first == RouteName.Hybrid) { var h = cache[q].H; return h.StructurallyOk ? (h.Answer, h.Tokens, h.Micros, false) : Escalate(q, h.Answer, h.Tokens, h.Micros); } var a = Get(q, first); if (a.StructurallyOk) return (a.Answer, a.Tokens, a.Micros, false); var b = Get(q, first == RouteName.Text ? RouteName.Graph : RouteName.Text); int tokens = a.Tokens + b.Tokens; double us = a.Micros + b.Micros; if (b.StructurallyOk) return (b.Answer, tokens, us, false); return Escalate(q, b.Answer ?? a.Answer, tokens, us); // the Ollama rung} ``` On seen phrasings, adding this ladder to the router took accuracy from 96.8% to 98.8%, which is the always-hybrid number, at 273 tokens per question instead of 573. That’s the same shape as Cekikj’s 83% to 91% at a lower cost than always-hybrid. Different data, different router, same lesson: check the result, not the router’s mood. Here’s the per-shape breakdown on the test set. “Silent” means the text route was wrong and every structural check passed. ``` TEST (seen phrasings): text route failures by question shape+-----------------+----+------------+-------------+--------------+---------------+| shape | n | text right | graph right | wrong+silent | wrong+flagged |+-----------------+----+------------+-------------+--------------+---------------+| Owner | 50 | 36/50 | 50/50 | 0 | 14 || Tier | 43 | 43/43 | 43/43 | 0 | 0 || OnCall | 44 | 44/44 | 0/44 | 0 | 0 || Define | 45 | 45/45 | 0/45 | 0 | 0 || IncidentService | 55 | 55/55 | 55/55 | 0 | 0 || ChainStores | 42 | 13/42 | 37/42 | 0 | 29 || TeamTierCount | 41 | 23/41 | 41/41 | 10 | 8 || TeamServices | 35 | 29/35 | 35/35 | 6 | 0 || NoRunbook | 45 | 13/45 | 45/45 | 32 | 0 |+-----------------+----+------------+-------------+--------------+---------------+ ``` Three things in that table taught me more than the headline numbers. **Chains get caught; absences don’t.** All 29 wrong chain answers were flagged, mostly by the dangling-reference check, because the reader could see it was following an edge into a page it didn’t have. But “which tier-1 services have no runbook?” was wrong 32 times out of 45 on the text route and not one of those failures raised a signal. The reader returned a perfectly well-formed list of the services it happened to retrieve. Absence and set questions fail by returning a plausible, smaller answer. No structural check I could think of catches that without knowing the full set, and if you know the full set you should have asked the graph. **The owner misses are a retrieval surprise.** “Which team owns X?” failed 14 times on text, because the words “team” and “owns” matched the team pages so strongly that BM25 ranked them above the service’s own page. Every one of those was flagged as a missing citation and recovered. I didn’t plan that failure mode, and I’d never have found it with a probability. **Stale text fails silently too.** The six TeamServices misses come from team pages that are about 10% out of date. The answer is cited, well formed, and wrong. On the shifted set the ladder helped much less: the router went from 84.0% to 85.3%. Most of its remaining errors were questions it sent to text that needed the graph, and they were exactly the silent kind. The last rung of the ladder is a stronger local model that gets both the retrieved passages and the graph rows, must cite what it uses, and may abstain: ``` public sealed class OllamaClient{ readonly HttpClient _http; public OllamaClient(string baseUrl) { _http = new HttpClient { BaseAddress = new Uri(baseUrl.TrimEnd('/') + "/"), Timeout = TimeSpan.FromMinutes(3) }; } public async Task<(JsonObject Json, int PromptTokens, int OutputTokens)> ChatJsonAsync( string model, string system, string user, JsonObject schema, CancellationToken ct = default) { var body = new JsonObject { ["model"] = model, ["stream"] = false, ["format"] = schema, ["options"] = new JsonObject { ["temperature"] = 0 }, ["messages"] = new JsonArray( new JsonObject { ["role"] = "system", ["content"] = system }, new JsonObject { ["role"] = "user", ["content"] = user }) }; using var resp = await _http.PostAsJsonAsync("api/chat", body, ct); resp.EnsureSuccessStatusCode(); var doc = JsonNode.Parse(await resp.Content.ReadAsStringAsync(ct))!; var content = doc["message"]?["content"]?.GetValue() ?? "{}"; var json = JsonNode.Parse(content) as JsonObject ?? throw new JsonException("model did not return a JSON object"); return (json, doc["prompt_eval_count"]?.GetValue() ?? 0, doc["eval_count"]?.GetValue() ?? 0); }}public sealed class OllamaEscalator(OllamaClient client, string model){ static readonly JsonObject Schema = JsonNode.Parse(""" {"type":"object","properties":{ "answer":{"type":"string"}, "citations":{"type":"array","items":{"type":"string"}}, "abstain":{"type":"boolean"}}, "required":["answer","citations","abstain"]} """)!.AsObject(); public async Task<(string? Answer, List Citations, int Tokens)> AnswerAsync( string question, string passages, string graphRows) { string user = $"Question: {question}\n\nWiki passages:\n{passages}\n\nGraph rows:\n{graphRows}\n\n" + "Answer only from the material above. Cite passage ids or \"graph\". " + "If the material is not enough, set abstain to true."; var (json, pIn, pOut) = await client.ChatJsonAsync(model, "You answer questions about internal payment services. Return JSON only.", user, Schema); var cites = json["citations"]?.AsArray().Select(x => x!.GetValue()).ToList() ?? new(); bool abstain = json["abstain"]?.GetValue() ?? true; // The same structural rule applies to the stronger model: no citation, no answer. string? answer = abstain || cites.Count == 0 ? null : json["answer"]?.GetValue(); return (answer, cites, pIn + pOut); }} ``` These need using System.Net.Http.Json;, using System.Text.Json; and using System.Text.Json.Nodes; at the top of the file. The router has an Ollama version too, with the same schema trick and a route enum of text, graph or hybrid plus a confidence number, so you can rerun my calibration check against a real model router. To run it locally with Docker: ``` docker run -d --name ollama -p 11434:11434 -v ollama:/root/.ollama ollama/ollamadocker exec ollama ollama pull qwen2.5:3bdocker exec ollama ollama pull qwen2.5:14bOLLAMA_URL=http://localhost:11434 \ROUTER_MODEL=qwen2.5:3b \ESCALATE_MODEL=qwen2.5:14b \dotnet run ``` Without OLLAMA_URL the program runs the deterministic replay and says plainly that the Ollama rung wasn't measured. Now the uncomfortable result: in my run, the rung would never have been called. Not once in 700 test and shifted questions did a question get past hybrid with a structural signal still firing. Every failure left after hybrid was silent. On the test set, hybrid was wrong 5 times. All 5 were chain questions where the graph had a missing call edge and returned a confident, well-formed, incomplete list of data stores. On the shifted set, hybrid was wrong 29 times, mostly on the new “what breaks if this store goes down” shape, where one missing edge anywhere upstream changes the answer. The graph got that shape right only 6 times out of 31. That is where the tautology bit me. I reran with the drift turned off (there’s a DRIFT environment variable; set it to 0) and the graph went to 42 of 42 on chains, 31 of 31 on blast-radius questions, and hybrid hit 100% on both sets. When the graph is exported from the same truth you grade against, the graph is always right. When it's 7% stale, the questions that look most like "obviously a graph question" degrade the fastest, because they touch the most edges. The only signal I found that catches any of those is disagreement: both routes answer cleanly and the answers differ. On the shifted set that happened 47 times, and the graph was the wrong one in 5 of them. So an “escalate on disagreement” rule would send 47 questions to the 14B model to catch 5 of the 29 hybrid errors. Whether that’s worth it depends on what a wrong blast-radius answer costs you. For an incident tool, probably yes. I haven’t measured what the 14B model does with those 47, and I’m not going to guess. All of this is my own replay on synthetic data, deterministic reader, no LLM in the loop. Tokens per question is chars divided by four of what a reader model would receive. p50 is in-process retrieval time in microseconds. ``` +--------------------------+----------+------------+-------------+-----------+-------------+-----------+| Route / strategy | test acc | test tok/q | test p50 us | shift acc | shift tok/q | fallbacks |+--------------------------+----------+------------+-------------+-----------+-------------+-----------+| always text | 75.2 % | 278 | 431 | 69.7 % | 280 | - || always graph | 76.5 % | 265 | 2 | 74.3 % | 270 | - || always hybrid | 98.8 % | 573 | 435 | 90.3 % | 580 | - || oracle (cheapest right) | 98.8 % | 256 | 11 | 90.3 % | 287 | - || NB router, argmax | 96.8 % | 268 | 376 | 84.0 % | 497 | - || NB + 0.6 fallback | 96.8 % | 274 | 378 | 84.0 % | 497 | 9 / 3 || NB + band (T on labels) | 98.3 % | 302 | 393 | 90.3 % | 580 | 49 / 91 || NB + band (T on correct) | 98.3 % | 307 | 393 | 90.3 % | 580 | 57 / 103 || NB + structural ladder | 98.8 % | 273 | 376 | 85.3 % | 507 | - || data-aware NB + ladder | 98.8 % | 295 | 381 | 90.3 % | 538 | - |+--------------------------+----------+------------+-------------+-----------+-------------+-----------+test = 400 seen phrasings, shift = 300 unseen phrasings.fallbacks = how often the confidence rule sent a question to hybrid (test / shift). ``` And Cekikj’s, for side-by-side reading, with the reminder that his are LLM routers on recorded gpt-4.1 outcomes and mine are not: ``` +---------------------------------+-------------+----------------+| Cekikj, 32 questions (author) | Accuracy | Cost/question |+---------------------------------+-------------+----------------+| always hybrid | 91% | 0.97 cents || oracle | 92% | 0.58 cents || 3 model routers | 83% | 0.63 to 0.81 c || router + empty-graph fallback | 91% | 0.75 cents || data-aware hint (20 questions) | 97.5% | 0.60 cents |+---------------------------------+-------------+----------------+ ``` **I’m keeping the router.** On phrasings it knew, it got within two points of always-hybrid at less than half the tokens, and with the structural ladder it matched hybrid at 48% of the cost. Routing is worth doing. **I’m not reading its confidence for anything that matters.** It was calibrated where it didn’t need to be and confidently wrong where it did. Calibration fitted on dev data didn’t survive a change in phrasing, and the version fitted on the wrong target made things worse. If I log the confidence at all, it’s to spot drift in its distribution, not to gate decisions. **I’m putting the fallbacks on observed results.** Empty result, missing citation, dangling reference, schema failure. They’re cheap, deterministic, and every one of them fired on real failures in my run. They also share a blind spot I now know the shape of: answers that are well formed and too small. Absence questions, set questions, and anything read from stale data. **I’m treating the graph’s correctness as a data freshness problem, not a retrieval problem.** Seven percent edge drift took the blast-radius shape from 31 of 31 to 6 of 31. Nothing in the routing layer can detect that, because the graph’s wrong answer is structurally perfect. What I still don’t know: If you build one of these, the cheapest useful thing you can do is the one I almost skipped: hold out a set of questions phrased in ways your router has never seen, and look at the confidence histogram on it. Mine looked fine until it didn’t. Tags: ai-agents, dotnet, csharp, rag, knowledge-graph, llm, ollama, calibration, model-routing, harness-engineering [Route First, Retrieve Second, But Never Trust the Router’s Confidence](https://pub.towardsai.net/route-first-retrieve-second-but-never-trust-the-routers-confidence-f86ac0367a3d) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.