Route First, Retrieve Second, But Never Trust the Router’s Confidence A replay of question-routing experiments by Miodrag Cekikj found that a confidence-based fallback threshold of 0.6 fired zero times across 150 routing decisions from three models, because gpt-4.1 reported 0.95 or higher on all 32 of its decisions (including six wrong ones), gpt-4o-mini never dropped below 0.70 across 50 decisions, and gpt-5-mini's lowest confidence was 0.86. An observed-signal rule instead — escalating to hybrid retrieval whenever the graph query returned empty or the agent answered without querying — fired on exactly the five trap questions and lifted accuracy from 83% to 91% at 0.75 cents per question versus 0.97 cents for always-hybrid, while a 242-token data-aware hint raised routers to 97.5% on 20 matrix questions. The author, who rebuilt the router in .NET and ran a calibration check, reports that the calibration fix made results worse, and cautions that Cekikj's 100% graph-traversal result against classic RAG's 13.6% on chain, set, absence and definition questions is near-tautological because the ground truth was generated from the same graph structure. The number that sent me down this hole was a zero. Miodrag Cekikj put a question router in front of three retrieval designs for an invented payments company he calls Naissus Pay. The router was a model with a strict structured output: pick a design, report a confidence, and if the confidence drops below 0.6, fall back to the expensive hybrid. A very normal setup. I have shipped almost exactly that config. Across 150 routing decisions from three different models, the 0.6 fallback fired zero times. Not because the routers were always right. They weren’t. gpt-4.1 reported 0.95 or higher on all 32 of its replayed decisions, including six wrong ones. gpt-4o-mini never went below 0.70 across 50 decisions, and its three costly misses all said 0.90. gpt-5-mini’s lowest confidence was 0.86. The safety net was hung at a height nothing ever fell to. That is Cekikj’s measurement on his own synthetic data, with his own caveats more on those below . I wanted to know whether it was a quirk of LLM routers or something more general, so I built my own router in .NET, gave it a confidence band, and ran a calibration check on a labelled set to prove or disprove that the confidence was usable. This post is what happened, including the part where my calibration fix made things worse. Every number in this section is Cekikj’s, from his “Route First, Retrieve Second” piece. He replayed strategies over 32 answered questions with recorded gpt-4.1 outcomes and computed ground truth, plus 18 routing-only questions written to test whether the router reads the shape of a question or just its keywords. +----------------------------------+----------+-------------------+| Strategy Cekikj, 32 questions | Accuracy | Cost per question |+----------------------------------+----------+-------------------+| Run all three, keep best | 92% | 1.99 cents || Oracle cheapest correct design | 92% | 0.58 cents || Always hybrid | 91% | 0.97 cents || Question routers 3 models | 83% | 0.63 to 0.81 c || Router + empty-graph fallback | 91% | 0.75 cents |+----------------------------------+----------+-------------------+Source: Miodrag Cekikj, author-run replay on synthetic data. Two results from that piece changed how I think about this. The first is the fallback that did work. Instead of trusting confidence, he escalated to hybrid whenever the graph query came back empty or the agent answered without querying. That rule fired on exactly the five trap questions with gpt-4o-mini routing and lifted accuracy from 83% to 91%, matching always-hybrid at 0.75 cents instead of 0.97. An observed result beat a reported probability. The second is the data-aware hint. A 242-token description of what the graph actually holds, added to the router prompt, took the routers to 97.5% on his 20 matrix questions once the missing structure had been built. He is upfront that the hint was written knowing which facts the trap questions needed, and that the escalation rule was replayed, not deployed. The sample is one invented company. I’d read 97.5% as “the router was starving for information about the data”, not as a number to expect. In a companion piece, “Not a Retrieval Problem, a Shape Problem”, Cekikj compares classic RAG with graph traversal on questions that are chains, sets, absences and definitions. Classic RAG scored 13.6% on those shapes. Graph traversal scored 100%. I want to be careful with that 100%. The ground truth was generated from the same structure the graph was built from. If the truth is computed by walking a graph, then walking that graph returns the truth. That isn’t a finding about graphs; it’s close to a tautology. What the experiment does show convincingly is the other half: top-k retrieval over passages cannot count, cannot enumerate a set it only partly retrieved, and cannot prove something is missing. That half doesn’t depend on how the truth was generated. His “Written Down or Worked Out” piece frames the same idea as two questions you ask before picking an architecture: is the answer written down or worked out, and does the knowledge live in text or in connected data? The nasty cell is “text, worked out”: the answer has to be assembled from many documents, and no retrieval trick fixes it. On his 20-question set, more passages and whole-corpus prompting both helped a little and cost a lot. Building the structure first is what moved the number. I ran into the tautology problem myself, and I’ll show you exactly where. Routers and gates look like different components. A router decides which path a question takes; a gate decides whether an action may run. Underneath, both are cheap classifiers whose output we treat as a decision, and both fail the same way: they can only rule on what reaches them, and they report confidence about things they never saw. Wasowski’s piece on agent gates and model routers makes this point with AmPermBench, which he describes as an independent stress test of Claude Code’s auto mode. In that benchmark, 36.8% of state changes 93 of 253 bypassed the permission classifier entirely, because they went through edits to files inside the project. During artifact cleanup the miss rate reached 92.9% 78 of 84 dangerous actions . His point that stuck with me: the classifier didn’t get those wrong, because nobody asked it. In his words, “What passes through the gate is still set by the harness.” The underlying paper is arXiv 2604.04978; I read it through Wasowski’s summary and a feed post, not end to end. Two other numbers have been circulating with Wasowski’s name attached: an 81% end-to-end miss rate, and an independent Jev router test of 825 samples at 51% accuracy. I couldn’t find either in his piece or trace them to a primary source, so I’m not leaning on them here. If you know where they come from, I’d like to read it. Part of why routers and gates are suddenly everywhere is that they got cheap. TypeSafe’s Jev and now Microsoft-Decision-1 are models that take a situation, a question and a fixed set of options, and return a probability per option. The pricing invites you to put one in front of every agent step. Before I did that, I sorted the numbers by who produced them. +--------------------------------------+--------------------------+-------------------------+| Claim | Number | Who measured it |+--------------------------------------+--------------------------+-------------------------+| Jev input price | $0.042 / 1M input tokens | TypeSafe vendor price || Jev latency | 70 to 500 ms | TypeSafe vendor || MS-Decision-1 input price | $0.042 / 1M, output free | Microsoft vendor || MS-Decision-1 speed vs GPT-6 Sol | about 35x faster P50 | Microsoft vendor || MS-Decision-1 accuracy | best on 36-bench set | Microsoft, own set || Jev phishing, one question | 62.6% Haiku 4.5: 81.3% | Independent repo || Jev phishing, split into five | 95.0% Haiku 4.5: 93.2% | Independent repo || Jev answers flipped by one injected | 12 to 22% | Braintrust, via || instruction | | Kalimatov's roundup || Simple Jev threshold router vs best | worse at 9 of 9 | rishirajFS, independent || single-model mix | operating points | |+--------------------------------------+--------------------------+-------------------------+ The independent results came mostly through Akzhan Kalimatov’s roundup of Jev tests. The phishing benchmark used 2,000 synthetic emails, and the five sub-questions were written after studying the dataset, so treat the 95% as an upper bound on what decomposition buys you. The Braintrust injection figure comes with no task, sample size or method in the roundup; I’d check the original before quoting it anywhere serious. The Microsoft-Decision-1 numbers are from Microsoft’s announcement as reported by Gowtham Boyina and others; at the time I looked, I found no third-party results. The pattern I take from the independent column is the same one Cekikj found. These models do well when the question is narrow and the options are described, and badly when one vague question carries all the weight. A third-party writeup I read made it concrete: a router given option names with no descriptions sent all 40 hard tasks to the cheap model at a median confidence of 0.96; one-line descriptions of each option fixed 37 of 40. Confidence was high in both runs. Only the information changed. The rishirajFS benchmark adds a sobering detail I’ll come back to: plain TF-IDF and small embedding routers, with no decision model at all, also beat single-model mixes there. The router doesn’t have to be fancy. It has to be checked. On cost, the NVIDIA researchers’ paper “Small Language Models are the Future of Agentic AI” is often quoted as saying agents overpay 10 to 30x. As Aqeel Abbas points out, it’s a position paper, and the 10 to 30x refers to serving a 7B model versus a 70B to 175B one. It’s an argument about serving cost, not a measured outcome for any agent. I wanted my own numbers, so I built a small replay harness in .NET 8 with no NuGet packages. It has: Now the honest part. The machine I ran this on couldn’t pull Ollama models, so no LLM is in the measured loop . The text route’s reader is deterministic: it parses only the passages BM25 retrieved and answers from those. That makes it an upper bound on a model reader’s comprehension, so every text-route failure you see below is a retrieval failure, not a reading failure. The router is a multinomial naive Bayes classifier over word unigrams and bigrams, trained on oracle labels. Naive Bayes is famously overconfident, which makes it a fair stand-in for “a cheap router that reports a probability”, but it is not gpt-4o-mini and I won’t pretend the numbers transfer. The Ollama router and the Ollama escalation rung are in the code and I tested the request and response handling against a stubbed HTTP handler. I didn’t measure them. Where a number would need a model, I’ll say so instead of filling it in. Costs are reported as tokens per question: the characters a reader model would receive, divided by four. Latency is in-process microseconds, which means it measures my retrieval code and nothing else. A real model call would dwarf it. I kept it in the table because the graph route’s 2 microseconds versus BM25’s 430 is still a true statement about the retrieval layer. I split the questions into 800 for training, 300 for dev, 400 for test using phrasings the router had seen, and 300 for a shifted set using phrasings it had never seen. The shifted set also includes one question shape the router never trained on: “if this data store goes down, what breaks, including indirectly?” The router interface is deliberately boring: public enum RouteName { Text, Graph, Hybrid }public record RouteDecision RouteName Route, double Confidence, IReadOnlyDictionary