Show HN: A model-routing benchmark – the routers optimize the wrong axis

A developer released a benchmark and tutorial series showing that model-routing systems optimize the wrong axis by routing ungrounded queries, while grounding—providing relevant context—dramatically improves accuracy across all model tiers, with a small local model plus retrieval outperforming an expensive ungrounded flagship model.

A model-routing benchmark and tutorial series: which model should handle a query — and is that even the right first question? Three model tiers, cold vs. grounded, scored with a deterministic citation-match metric against the published code text. 45 real questions about the 2023 8th-Edition Florida Building Code and the Naples/Collier County local amendments, each with a gold section citation verified against the published text. Every number below regenerates from the committed raw results via analysis/benchmark report.ipynb /dotnetspark/fbc-model-routing-benchmark/blob/main/analysis/benchmark report.ipynb . → Full synthesis with method, caveats, and the router study: FINDINGS.md | tier | cold — correct / wrong / abstain | grounded — correct / wrong / abstain | Δ correct | |---|---|---|---| | Claude Opus 4.8 foundation role | 26.5% / 31.8% / 41.7% | 59.1% / 11.4% / 29.5% | +32.6 | | Claude Haiku 4.5 instruction-tuned, forced JSON schema | 31.8% / 67.4% / 0.8% | 77.3% / 22.7% / 0.0% | +45.5 | | phi3:mini ~3.8B SLM, local Ollama | 7.9% / 39.4% / 52.8% | 52.5% / 12.5% / 35.0% | +44.6 | Cold = parametric memory only, 45 questions × 3 repeats scoreable n = 132/132/127 . Grounded = the correct code passage injected as context, single pass n = 44/44/40 . "Wrong" = cited a section the published code doesn't assign — a hallucinated citation. Three findings carry the series: Cold, no tier is usable — and each fails differently. The flagship abstains 42%, calibrated honesty , the schema-forced cheap model fabricates 67% wrong — its required section field forbids "I don't know" , and the small local model simply doesn't know 7.9% correct . A schema guarantees a parseable answer, never a correct one. Grounding rescues the cheap local model. Retrieval lifts every tier by 33–46 points; grounded phi3 52.5%, on a laptop, ~free, offline beats cold Opus 26.5% by ~2×. Cheap local model + good retrieval beats an expensive model alone. Grounding is a separate axis that belongs Off the shelf, every router we dry-ran RouteLLM, NotDiamond routes 100% ungrounded — not because they're bad, but because grounding is a pipeline decision outside the "pick a model" abstraction. Ground the prompt yourself and in front of model routing. train NotDiamond's custom router on your own scores, and it lands cheap + grounded on its own — matching the hand-built router: Gemini's free tier covers the whole benchmark; no paid key needed. git clone https://github.com/dotnetspark/fbc-model-routing-benchmark.git cd fbc-model-routing-benchmark pip install -r requirements.txt Python 3.11+ Free key from https://aistudio.google.com → put it in .env: GEMINI API KEY=... python run benchmark.py --model-class foundation gemini --repeats 1 --limit 5 smoke test python run benchmark.py --model-class foundation gemini full cold run, $0 Each run writes one JSON line per request to results/<model class raw.jsonl and prints a latency/cost/citation summary. A result line looks like: {"model class": "foundation", "model name": "claude-opus-4-8", "question id": "q001", "output": "…", "latency ms": 7157.1, "input tokens": 45, "output tokens": 467, "cost usd": 0.0119, "cited section": "10", "error": null} Then regenerate every chart and table from the raw results: jupyter nbconvert --to notebook --execute --inplace analysis/benchmark report.ipynb Each lesson: Concept → why it matters for routing → build increment → measured checkpoint the checkpoint shows the reference run's actual numbers, so you know what you should see . Work through them in order. | lesson | what you build & measure | time | API cost | |---|---|---|---| | 02 — Instruction-tuned models /dotnetspark/fbc-model-routing-benchmark/blob/main/lessons/02-instruction-tuned-models.md 03 — Small language models /dotnetspark/fbc-model-routing-benchmark/blob/main/lessons/03-small-language-models.md 04 — RAG-integrated models /dotnetspark/fbc-model-routing-benchmark/blob/main/lessons/04-rag-integrated-models.md 05 — When to use each model type /dotnetspark/fbc-model-routing-benchmark/blob/main/lessons/05-when-to-use-each-model-type.md Multimodal site plans, diagrams . "Does this site plan meet the front-setback requirement?" is a real FBC workload, but it needs its own metric did the model read the dimension actually on the drawing? and its own gold set — bolting a handful of image questions onto a 45-question citation benchmark would produce directional-only numbers. Measured properly, it's a separate project. Fine-tuning. The interesting question is economic, not technical: at what request volume does a one-time tuning cost beat the per-request cost of a bigger or grounded model? Given the series' central finding — grounding, not model tier, is the dominant lever, and grounding requires no training — fine-tuning is the third thing to reach for, after retrieval and tier routing. A break-even template with placeholder inputs lives in the notebook Section 4 ; no finding in the series depends on it. ├── README.md ← results overview + quickstart you are here ├── FINDINGS.md ← the full synthesis — start here for the analysis ├── requirements.txt ├── run benchmark.py ← the only place inference happens Lessons 1–4 ├── run router dryrun.py ← Lesson 5: records each router's selections ├── run notdiamond training.py ← Lesson 5: trains NotDiamond's custom router on our scores ├── docker/ docker-compose.yml ← Linux env for the router stack RouteLLM won't install on Windows ├── lessons/ ← the tutorial series 01–05 ├── clients/ ← model clients + the shared regex citation extractor the metric ├── routers/ ← router adapters: custom lookup, RouteLLM, NotDiamond, heuristic ├── data/ ← gold questions, verified grounding excerpts, source corpus PDFs ├── results/ ← committed raw results .jsonl , charts, summary tables └── analysis/ ← benchmark report.ipynb regenerates every number + router comparison - The Florida Building Code and the Naples/Collier local amendments are public documents with clear, checkable answers — so the metric can be a deterministic regex section-citation match , not an LLM judge. Correct / wrong / abstained are unambiguous. - Code text is dense, cross-referenced, and frequently amended — a genuine stress test for parametric memory, grounding, and hallucination detection. The local-jurisdiction questions 22 of 45 are the narrowest, least-trained-on material, which is exactly where the interesting failures live. - The end state is a reusable evaluation harness: fork it, point it at your own jurisdiction's code, publish your own numbers. pip install -r requirements.txt Cold runs Lessons 1–3 . Keys in .env: ANTHROPIC API KEY and/or GEMINI API KEY. python run benchmark.py --model-class foundation Opus 4.8 paid python run benchmark.py --model-class foundation gemini Gemini free tier, $0 python run benchmark.py --model-class instruction tuned Haiku 4.5 + JSON schema python run benchmark.py --model-class slm phi3:mini via local Ollama Grounded runs Lesson 4 — needs data/fbc eval context.csv committed . python run benchmark.py --model-class foundation --grounded python run benchmark.py --model-class instruction tuned --grounded python run benchmark.py --model-class slm --grounded Router dry-run Lesson 5 — Linux container because RouteLLM won't install on Windows. docker compose run --rm routers Optional: NotDiamond custom-router training Lesson 5; NOT DIAMOND API KEY in .env . docker compose run --rm routers python run notdiamond training.py Regenerate every number and chart from the raw results. jupyter nbconvert --to notebook --execute --inplace analysis/benchmark report.ipynb Copyright note: the FBC text is ICC-copyrighted. Only short per-question excerpts ship in data/fbc eval context.csv /dotnetspark/fbc-model-routing-benchmark/blob/main/data/fbc eval context.csv ; reproduce full passages from the published sources see Lesson 4 . This is a research/benchmarking tool, not a substitute for a licensed design professional or the Naples/Collier County building department. Do not make compliance decisions from it — the benchmark's own headline result is that no configuration tested is reliable enough for that. MIT — fork it, point it at your own jurisdiction's code, publish your own numbers.