A model-routing benchmark and tutorial series: which model should handle a query — and is that even the right first question? Three model tiers, cold vs. grounded, scored with a deterministic citation-match metric against the published code text.
45 real questions about the 2023 8th-Edition Florida Building Code and the Naples/Collier County local amendments, each with a gold section citation verified against the published text. Every number below regenerates from the committed raw results via analysis/benchmark_report.ipynb.
→ Full synthesis with method, caveats, and the router study: FINDINGS.md
| tier | cold — correct / wrong / abstain | grounded — correct / wrong / abstain | Δ correct |
|---|---|---|---|
| Claude Opus 4.8 (foundation role) | 26.5% / 31.8% / 41.7% | 59.1% / 11.4% / 29.5% | |
| +32.6 | |||
| Claude Haiku 4.5 (instruction-tuned, forced JSON schema) | 31.8% / 67.4% / 0.8% | 77.3% / 22.7% / 0.0% | |
| +45.5 | |||
| phi3:mini (~3.8B SLM, local Ollama) | 7.9% / 39.4% / 52.8% | 52.5% / 12.5% / 35.0% | |
| +44.6 |
Cold = parametric memory only, 45 questions × 3 repeats (scoreable n = 132/132/127). Grounded = the correct code passage injected as context, single pass (n = 44/44/40). "Wrong" = cited a section the published code doesn't assign — a hallucinated citation.
Three findings carry the series:
Cold, no tier is usable — and each fails differently. The flagshipabstains(42%, calibrated honesty), the schema-forced cheap modelfabricates(67% wrong — its requiredsection
field forbids "I don't know"), and the small local model simplydoesn't know(7.9% correct). A schema guarantees a parseable answer, never a correct one.Grounding rescues the cheap local model. Retrieval lifts every tier by 33–46 points; grounded phi3 (52.5%, on a laptop, ~free, offline) beatscoldOpus (26.5%) by ~2×.**Cheap local model + good retrieval beats an expensive model alone.Grounding is a separate axis that belongs Off the shelf, every router we dry-ran (RouteLLM, NotDiamond) routes 100% ungrounded — not because they're bad, but because grounding is a pipeline decision outside the "pick a model" abstraction. Ground the prompt yourself andin front ofmodel routing.trainNotDiamond's custom router on your own scores, and it lands cheap + grounded on its own — matching the hand-built router:
Gemini's free tier covers the whole benchmark; no paid key needed.
git clone https://github.com/dotnetspark/fbc-model-routing-benchmark.git
cd fbc-model-routing-benchmark
pip install -r requirements.txt # Python 3.11+
python run_benchmark.py --model-class foundation_gemini --repeats 1 --limit 5 # smoke test
python run_benchmark.py --model-class foundation_gemini # full cold run, $0
Each run writes one JSON line per request to results/<model_class>_raw.jsonl
and prints a latency/cost/citation summary. A result line looks like:
{"model_class": "foundation", "model_name": "claude-opus-4-8", "question_id": "q001",
"output": "…", "latency_ms": 7157.1, "input_tokens": 45, "output_tokens": 467,
"cost_usd": 0.0119, "cited_section": "10", "error": null}
Then regenerate every chart and table from the raw results:
jupyter nbconvert --to notebook --execute --inplace analysis/benchmark_report.ipynb
Each lesson: Concept → why it matters for routing → build increment → measured checkpoint (the checkpoint shows the reference run's actual numbers, so you know what you should see). Work through them in order.
| lesson | what you build & measure | time | API cost |
|---|---|---|---|
02 — Instruction-tuned models03 — Small language models04 — RAG-integrated models05 — When to use each model typeMultimodal (site plans, diagrams)."Does this site plan meet the front-setback requirement?" is a real FBC workload, but it needs its own metric (did the model read the dimension actually on the drawing?) and its own gold set — bolting a handful of image questions onto a 45-question citation benchmark would produce directional-only numbers. Measured properly, it's a separate project.Fine-tuning. The interesting question is economic, not technical: at what request volume does a one-time tuning cost beat the per-request cost of a bigger (or grounded) model? Given the series' central finding — grounding, not model tier, is the dominant lever, and grounding requires no training — fine-tuning is thethirdthing to reach for, after retrieval and tier routing. A break-even template with placeholder inputs lives in the notebook (Section 4); no finding in the series depends on it.
├── README.md ← results overview + quickstart (you are here)
├── FINDINGS.md ← the full synthesis — start here for the analysis
├── requirements.txt
├── run_benchmark.py ← the only place inference happens (Lessons 1–4)
├── run_router_dryrun.py ← Lesson 5: records each router's selections
├── run_notdiamond_training.py ← Lesson 5: trains NotDiamond's custom router on our scores
├── docker/ docker-compose.yml ← Linux env for the router stack (RouteLLM won't install on Windows)
├── lessons/ ← the tutorial series (01–05)
├── clients/ ← model clients + the shared regex citation extractor (the metric)
├── routers/ ← router adapters: custom lookup, RouteLLM, NotDiamond, heuristic
├── data/ ← gold questions, verified grounding excerpts, source corpus PDFs
├── results/ ← committed raw results (.jsonl), charts, summary tables
└── analysis/ ← benchmark_report.ipynb (regenerates every number) + router comparison
- The Florida Building Code and the Naples/Collier local amendments are public documents with clear, checkable answers — so the metric can be a deterministic regex section-citation match, not an LLM judge. Correct / wrong / abstained are unambiguous. - Code text is dense, cross-referenced, and frequently amended — a genuine stress test for parametric memory, grounding, and hallucination detection. The local-jurisdiction questions (22 of 45) are the narrowest, least-trained-on material, which is exactly where the interesting failures live.
- The end state is a reusable evaluation harness: fork it, point it at your own jurisdiction's code, publish your own numbers.
pip install -r requirements.txt
python run_benchmark.py --model-class foundation # Opus 4.8 (paid)
python run_benchmark.py --model-class foundation_gemini # Gemini free tier, $0
python run_benchmark.py --model-class instruction_tuned # Haiku 4.5 + JSON schema
python run_benchmark.py --model-class slm # phi3:mini via local Ollama
python run_benchmark.py --model-class foundation --grounded
python run_benchmark.py --model-class instruction_tuned --grounded
python run_benchmark.py --model-class slm --grounded
docker compose run --rm routers
docker compose run --rm routers python run_notdiamond_training.py
jupyter nbconvert --to notebook --execute --inplace analysis/benchmark_report.ipynb
Copyright note: the FBC text is ICC-copyrighted. Only short per-question excerpts ship in data/fbc_eval_context.csv; reproduce full passages from the published sources (see Lesson 4).
This is a research/benchmarking tool, not a substitute for a licensed design professional or the Naples/Collier County building department. Do not make compliance decisions from it — the benchmark's own headline result is that no configuration tested is reliable enough for that.
MIT — fork it, point it at your own jurisdiction's code, publish your own numbers.