There's a pattern I see every release cycle: a new budget-friendly model ships, the discourse explodes with hot takes, and within 48 hours half my feed has declared it a drop-in replacement for everything. The claim might even be true. But here's what nobody posting those takes can tell you: whether it's true for your specific workload. And most of the time, that's the only question that matters.
A healthier mental model: stop treating model selection as a one-time shopping decision and start treating it as a runtime policy. Route work to the inexpensive option by default, check the output with something that isn't a model, and only pay for the heavyweight option when the check fails. Below is a working implementation of that policy, plus the measurement discipline that turns it from a hunch into an auditable system.
My configuration is deliberately boring:
On the cost side, one note: I iterate on this pipeline using MonkeyCode, which at the time of writing provides free model access along with a free server option, so experimentation doesn't rack up a bill. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The pipeline itself is provider-agnostic — anything speaking the OpenAI-compatible chat API drops in, so treat endpoints as configuration, not commitment.
The single most important design decision: never let a model decide whether its own output (or a peer's) was good enough. LLMs report confidence regardless of correctness. Instead, escalate only when an external, deterministic check fails — a test suite, a parser, a schema validator, a diff against expected output.
Here's a compact implementation. Standard library plus requests
, fully rerunnable, and every routing decision gets written to a JSONL audit log:
import hashlib, json, subprocess, time
from dataclasses import dataclass, asdict
import requests
@dataclass
class RouteRecord:
job_id: str
lane: str # "A" or "B"
fell_back: bool
check_ok: bool
seconds: float
prompt_fingerprint: str
def chat(endpoint: str, model: str, prompt: str) -> str:
resp = requests.post(
f"{endpoint}/v1/chat/completions",
json={
"model": model,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.2,
},
timeout=120,
)
resp.raise_for_status()
return resp.json()["choices"][0]["message"]["content"]
def objective_check(job: dict, candidate: str) -> bool:
"""Deterministic validation only. No LLM-as-judge allowed here."""
mode = job["check"]
if mode == "pytest":
path = job["write_to"]
with open(path, "w") as fh:
fh.write(strip_fences(candidate))
run = subprocess.run(job["command"], shell=True,
capture_output=True, timeout=90)
return run.returncode == 0
if mode == "valid_json":
try:
parsed = json.loads(strip_fences(candidate))
except json.JSONDecodeError:
return False
required = job.get("required_keys", [])
return all(k in parsed for k in required)
if mode == "regex":
import re
return re.fullmatch(job["pattern"], candidate.strip()) is not None
raise ValueError(f"unsupported check: {mode}")
def strip_fences(text: str) -> str:
"""Pull code out of a markdown fence if present; else return as-is."""
if "```
" not in text:
return text
block = text.split("
```")[1]
lines = block.splitlines()
if lines and lines[0].strip().isalpha(): # language tag line
lines = lines[1:]
return "\n".join(lines)
def route(job: dict, lanes: list[dict], log_path: str = "routes.jsonl") -> RouteRecord:
fp = hashlib.sha256(job["prompt"].encode()).hexdigest()[:12]
fell_back = False
for idx, lane in enumerate(lanes):
start = time.time()
candidate = chat(lane["endpoint"], lane["model"], job["prompt"])
elapsed = round(time.time() - start, 2)
passed = objective_check(job, candidate)
if passed or idx == len(lanes) - 1:
record = RouteRecord(
job_id=job["id"],
lane=lane["label"],
fell_back=fell_back,
check_ok=passed,
seconds=elapsed,
prompt_fingerprint=fp,
)
with open(log_path, "a") as fh:
fh.write(json.dumps(asdict(record)) + "\n")
return record
fell_back = True
Wiring it up:
lanes = [
{"label": "A", "endpoint": "https://lane-a-endpoint", "model": "current-budget-model"},
{"label": "B", "endpoint": "https://lane-b-endpoint", "model": "premium-model"},
]
job = {
"id": "csv-to-json-migration-014",
"prompt": (
"Convert the transformation in migrate.py so it emits newline-delimited JSON. "
"Return only the complete updated file inside a code fence."
),
"check": "pytest",
"write_to": "migrate.py",
"command": "python -m pytest tests/test_migrate.py -q",
}
print(route(job, lanes))
The router code is maybe a weekend of effort. The compounding value lives in routes.jsonl
. After a few weeks of real traffic you can compute things that are otherwise pure speculation:
fell_back
. If data-formatting jobs pass on Lane A 92% of the time, Lane A is a rational default there. If multi-file refactors fail 60% of the time, Lane A is a false economy for that family — you're paying for a doomed first attempt plus added latency on most calls.Operating rules I'd insist on:
If most of your workload is unverifiable generation, or everything you run is on a hard latency budget, honestly — skip the router. Picking the strong model outright is the simpler and more correct engineering decision in that world.
Pull your last ~50 real prompts, sort them into "has a deterministic check" versus "doesn't," and push the checkable subset through the two-lane setup for a week. If you want the measurement phase to cost nothing, MonkeyCode's free model access and free server option work fine as Lane A and host while you gather data — and since the log format is provider-neutral, whatever you learn transfers when you point the lanes elsewhere.
The question "is the cheap model good enough?" has an answer, and it's already sitting in your prompt history. Measure it; don't outsource the decision to launch-week sentiment.