{"slug": "show-hn-a-model-routing-benchmark-the-routers-optimize-the-wrong-axis", "title": "Show HN: A model-routing benchmark – the routers optimize the wrong axis", "summary": "A developer released a benchmark and tutorial series showing that model-routing systems optimize the wrong axis by routing ungrounded queries, while grounding—providing relevant context—dramatically improves accuracy across all model tiers, with a small local model plus retrieval outperforming an expensive ungrounded flagship model.", "body_md": "**A model-routing benchmark and tutorial series: which model should handle a query — and is that even the right first question? Three model tiers, cold vs. grounded, scored with a deterministic citation-match metric against the published code text.**\n\n45 real questions about the 2023 8th-Edition Florida Building Code and the Naples/Collier County local amendments, each with a gold section citation verified against the published text. Every number below regenerates from the committed raw results via [ analysis/benchmark_report.ipynb](/dotnetspark/fbc-model-routing-benchmark/blob/main/analysis/benchmark_report.ipynb).\n\n**→ Full synthesis with method, caveats, and the router study: FINDINGS.md**\n\n| tier | cold — correct / wrong / abstain | grounded — correct / wrong / abstain | Δ correct |\n|---|---|---|---|\n| Claude Opus 4.8 (foundation role) | 26.5% / 31.8% / 41.7% | 59.1% / 11.4% / 29.5% |\n+32.6 |\n| Claude Haiku 4.5 (instruction-tuned, forced JSON schema) | 31.8% / 67.4% / 0.8% | 77.3% / 22.7% / 0.0% |\n+45.5 |\n| phi3:mini (~3.8B SLM, local Ollama) | 7.9% / 39.4% / 52.8% | 52.5% / 12.5% / 35.0% |\n+44.6 |\n\n*Cold = parametric memory only, 45 questions × 3 repeats (scoreable n = 132/132/127). Grounded = the correct code passage injected as context, single pass (n = 44/44/40). \"Wrong\" = cited a section the published code doesn't assign — a hallucinated citation.*\n\nThree findings carry the series:\n\n**Cold, no tier is usable — and each fails differently.** The flagship*abstains*(42%, calibrated honesty), the schema-forced cheap model*fabricates*(67% wrong — its required`section`\n\nfield forbids \"I don't know\"), and the small local model simply*doesn't know*(7.9% correct). A schema guarantees a parseable answer, never a correct one.**Grounding rescues the cheap local model.** Retrieval lifts every tier by 33–46 points; grounded phi3 (52.5%, on a laptop, ~free, offline) beats*cold*Opus (26.5%) by ~2×.**Cheap local model + good retrieval beats an expensive model alone.****Grounding is a separate axis that belongs** Off the shelf, every router we dry-ran (RouteLLM, NotDiamond) routes 100% ungrounded — not because they're bad, but because grounding is a pipeline decision outside the \"pick a model\" abstraction. Ground the prompt yourself and*in front of*model routing.*train*NotDiamond's custom router on your own scores, and it lands cheap + grounded on its own — matching the hand-built router:\n\nGemini's free tier covers the whole benchmark; no paid key needed.\n\n```\ngit clone https://github.com/dotnetspark/fbc-model-routing-benchmark.git\ncd fbc-model-routing-benchmark\npip install -r requirements.txt            # Python 3.11+\n\n# Free key from https://aistudio.google.com → put it in .env:\n#   GEMINI_API_KEY=...\n\npython run_benchmark.py --model-class foundation_gemini --repeats 1 --limit 5   # smoke test\npython run_benchmark.py --model-class foundation_gemini                          # full cold run, $0\n```\n\nEach run writes one JSON line per request to `results/<model_class>_raw.jsonl`\n\nand prints a latency/cost/citation summary. A result line looks like:\n\n```\n{\"model_class\": \"foundation\", \"model_name\": \"claude-opus-4-8\", \"question_id\": \"q001\",\n \"output\": \"…\", \"latency_ms\": 7157.1, \"input_tokens\": 45, \"output_tokens\": 467,\n \"cost_usd\": 0.0119, \"cited_section\": \"10\", \"error\": null}\n```\n\nThen regenerate every chart and table from the raw results:\n\n```\njupyter nbconvert --to notebook --execute --inplace analysis/benchmark_report.ipynb\n```\n\nEach lesson: **Concept → why it matters for routing → build increment → measured checkpoint** (the checkpoint shows the reference run's actual numbers, so you know what you should see). Work through them in order.\n\n| lesson | what you build & measure | time | API cost |\n|---|---|---|---|\n|\n\n[02 — Instruction-tuned models](/dotnetspark/fbc-model-routing-benchmark/blob/main/lessons/02-instruction-tuned-models.md)[03 — Small language models](/dotnetspark/fbc-model-routing-benchmark/blob/main/lessons/03-small-language-models.md)[04 — RAG-integrated models](/dotnetspark/fbc-model-routing-benchmark/blob/main/lessons/04-rag-integrated-models.md)[05 — When to use each model type](/dotnetspark/fbc-model-routing-benchmark/blob/main/lessons/05-when-to-use-each-model-type.md)**Multimodal (site plans, diagrams).**\"Does this site plan meet the front-setback requirement?\" is a real FBC workload, but it needs its own metric (did the model read the dimension actually on the drawing?) and its own gold set — bolting a handful of image questions onto a 45-question citation benchmark would produce directional-only numbers. Measured properly, it's a separate project.**Fine-tuning.** The interesting question is economic, not technical: at what request volume does a one-time tuning cost beat the per-request cost of a bigger (or grounded) model? Given the series' central finding — grounding, not model tier, is the dominant lever, and grounding requires no training — fine-tuning is the*third*thing to reach for, after retrieval and tier routing. A break-even template with placeholder inputs lives in the notebook (Section 4); no finding in the series depends on it.\n\n```\n├── README.md                    ← results overview + quickstart (you are here)\n├── FINDINGS.md                  ← the full synthesis — start here for the analysis\n├── requirements.txt\n├── run_benchmark.py             ← the only place inference happens (Lessons 1–4)\n├── run_router_dryrun.py         ← Lesson 5: records each router's selections\n├── run_notdiamond_training.py   ← Lesson 5: trains NotDiamond's custom router on our scores\n├── docker/  docker-compose.yml  ← Linux env for the router stack (RouteLLM won't install on Windows)\n├── lessons/                     ← the tutorial series (01–05)\n├── clients/                     ← model clients + the shared regex citation extractor (the metric)\n├── routers/                     ← router adapters: custom lookup, RouteLLM, NotDiamond, heuristic\n├── data/                        ← gold questions, verified grounding excerpts, source corpus PDFs\n├── results/                     ← committed raw results (.jsonl), charts, summary tables\n└── analysis/                    ← benchmark_report.ipynb (regenerates every number) + router comparison\n```\n\n- The Florida Building Code and the Naples/Collier local amendments are public documents with clear, checkable answers — so the metric can be a\n**deterministic regex section-citation match**, not an LLM judge. Correct / wrong / abstained are unambiguous. - Code text is dense, cross-referenced, and frequently amended — a genuine stress test for parametric memory, grounding, and hallucination detection. The local-jurisdiction questions (22 of 45) are the narrowest, least-trained-on material, which is exactly where the interesting failures live.\n- The end state is a reusable evaluation harness: fork it, point it at your own jurisdiction's code, publish your own numbers.\n\n```\npip install -r requirements.txt\n\n# Cold runs (Lessons 1–3). Keys in .env: ANTHROPIC_API_KEY and/or GEMINI_API_KEY.\npython run_benchmark.py --model-class foundation                 # Opus 4.8 (paid)\npython run_benchmark.py --model-class foundation_gemini          # Gemini free tier, $0\npython run_benchmark.py --model-class instruction_tuned          # Haiku 4.5 + JSON schema\npython run_benchmark.py --model-class slm                        # phi3:mini via local Ollama\n\n# Grounded runs (Lesson 4) — needs data/fbc_eval_context.csv (committed).\npython run_benchmark.py --model-class foundation --grounded\npython run_benchmark.py --model-class instruction_tuned --grounded\npython run_benchmark.py --model-class slm --grounded\n\n# Router dry-run (Lesson 5) — Linux container because RouteLLM won't install on Windows.\ndocker compose run --rm routers\n\n# Optional: NotDiamond custom-router training (Lesson 5; NOT_DIAMOND_API_KEY in .env).\ndocker compose run --rm routers python run_notdiamond_training.py\n\n# Regenerate every number and chart from the raw results.\njupyter nbconvert --to notebook --execute --inplace analysis/benchmark_report.ipynb\n```\n\n**Copyright note:** the FBC text is ICC-copyrighted. Only short per-question excerpts ship in [ data/fbc_eval_context.csv](/dotnetspark/fbc-model-routing-benchmark/blob/main/data/fbc_eval_context.csv); reproduce full passages from the published sources (see Lesson 4).\n\nThis is a research/benchmarking tool, **not** a substitute for a licensed design professional or the Naples/Collier County building department. Do not make compliance decisions from it — the benchmark's own headline result is that no configuration tested is reliable enough for that.\n\nMIT — fork it, point it at your own jurisdiction's code, publish your own numbers.", "url": "https://wpnews.pro/news/show-hn-a-model-routing-benchmark-the-routers-optimize-the-wrong-axis", "canonical_source": "https://github.com/dotnetspark/fbc-model-routing-benchmark", "published_at": "2026-07-18 04:55:51+00:00", "updated_at": "2026-07-18 05:21:10.513580+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "developer-tools"], "entities": ["Claude Opus", "Claude Haiku", "phi3", "RouteLLM", "NotDiamond", "Gemini", "Ollama"], "alternates": {"html": "https://wpnews.pro/news/show-hn-a-model-routing-benchmark-the-routers-optimize-the-wrong-axis", "markdown": "https://wpnews.pro/news/show-hn-a-model-routing-benchmark-the-routers-optimize-the-wrong-axis.md", "text": "https://wpnews.pro/news/show-hn-a-model-routing-benchmark-the-routers-optimize-the-wrong-axis.txt", "jsonld": "https://wpnews.pro/news/show-hn-a-model-routing-benchmark-the-routers-optimize-the-wrong-axis.jsonld"}}