Cheap Model First, Strong Model on Failure: Building an Auditable Two-Tier LLM Pipeline

A developer has built an auditable two-tier LLM pipeline that routes work to a cheap model by default and escalates to a stronger model only when an external deterministic check fails. The implementation, written in Python using only the standard library and requests, logs every routing decision to a JSONL audit file. The design explicitly avoids using an LLM as a judge, relying instead on test suites, JSON schema validators, or regex patterns for validation.

There's a pattern I see every release cycle: a new budget-friendly model ships, the discourse explodes with hot takes, and within 48 hours half my feed has declared it a drop-in replacement for everything. The claim might even be true. But here's what nobody posting those takes can tell you: whether it's true for your specific workload . And most of the time, that's the only question that matters. A healthier mental model: stop treating model selection as a one-time shopping decision and start treating it as a runtime policy. Route work to the inexpensive option by default, check the output with something that isn't a model, and only pay for the heavyweight option when the check fails. Below is a working implementation of that policy, plus the measurement discipline that turns it from a hunch into an auditable system. My configuration is deliberately boring: On the cost side, one note: I iterate on this pipeline using MonkeyCode, which at the time of writing provides free model access along with a free server option, so experimentation doesn't rack up a bill. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The pipeline itself is provider-agnostic — anything speaking the OpenAI-compatible chat API drops in, so treat endpoints as configuration, not commitment. The single most important design decision: never let a model decide whether its own output or a peer's was good enough. LLMs report confidence regardless of correctness. Instead, escalate only when an external, deterministic check fails — a test suite, a parser, a schema validator, a diff against expected output. Here's a compact implementation. Standard library plus requests , fully rerunnable, and every routing decision gets written to a JSONL audit log: python lane router.py import hashlib, json, subprocess, time from dataclasses import dataclass, asdict import requests @dataclass class RouteRecord: job id: str lane: str "A" or "B" fell back: bool check ok: bool seconds: float prompt fingerprint: str def chat endpoint: str, model: str, prompt: str - str: resp = requests.post f"{endpoint}/v1/chat/completions", json={ "model": model, "messages": {"role": "user", "content": prompt} , "temperature": 0.2, }, timeout=120, resp.raise for status return resp.json "choices" 0 "message" "content" def objective check job: dict, candidate: str - bool: """Deterministic validation only. No LLM-as-judge allowed here.""" mode = job "check" if mode == "pytest": path = job "write to" with open path, "w" as fh: fh.write strip fences candidate run = subprocess.run job "command" , shell=True, capture output=True, timeout=90 return run.returncode == 0 if mode == "valid json": try: parsed = json.loads strip fences candidate except json.JSONDecodeError: return False required = job.get "required keys", return all k in parsed for k in required if mode == "regex": import re return re.fullmatch job "pattern" , candidate.strip is not None raise ValueError f"unsupported check: {mode}" def strip fences text: str - str: """Pull code out of a markdown fence if present; else return as-is.""" if " " not in text: return text block = text.split " " 1 lines = block.splitlines if lines and lines 0 .strip .isalpha : language tag line lines = lines 1: return "\n".join lines def route job: dict, lanes: list dict , log path: str = "routes.jsonl" - RouteRecord: fp = hashlib.sha256 job "prompt" .encode .hexdigest :12 fell back = False for idx, lane in enumerate lanes : start = time.time candidate = chat lane "endpoint" , lane "model" , job "prompt" elapsed = round time.time - start, 2 passed = objective check job, candidate if passed or idx == len lanes - 1: record = RouteRecord job id=job "id" , lane=lane "label" , fell back=fell back, check ok=passed, seconds=elapsed, prompt fingerprint=fp, with open log path, "a" as fh: fh.write json.dumps asdict record + "\n" return record fell back = True Wiring it up: lanes = {"label": "A", "endpoint": "https://lane-a-endpoint", "model": "current-budget-model"}, {"label": "B", "endpoint": "https://lane-b-endpoint", "model": "premium-model"}, job = { "id": "csv-to-json-migration-014", "prompt": "Convert the transformation in migrate.py so it emits newline-delimited JSON. " "Return only the complete updated file inside a code fence." , "check": "pytest", "write to": "migrate.py", "command": "python -m pytest tests/test migrate.py -q", } print route job, lanes The router code is maybe a weekend of effort. The compounding value lives in routes.jsonl . After a few weeks of real traffic you can compute things that are otherwise pure speculation: fell back . If data-formatting jobs pass on Lane A 92% of the time, Lane A is a rational default there. If multi-file refactors fail 60% of the time, Lane A is a false economy for that family — you're paying for a doomed first attempt plus added latency on most calls.Operating rules I'd insist on: If most of your workload is unverifiable generation, or everything you run is on a hard latency budget, honestly — skip the router. Picking the strong model outright is the simpler and more correct engineering decision in that world. Pull your last ~50 real prompts, sort them into "has a deterministic check" versus "doesn't," and push the checkable subset through the two-lane setup for a week. If you want the measurement phase to cost nothing, MonkeyCode's free model access and free server option work fine as Lane A and host while you gather data — and since the log format is provider-neutral, whatever you learn transfers when you point the lanes elsewhere. The question "is the cheap model good enough?" has an answer, and it's already sitting in your prompt history. Measure it; don't outsource the decision to launch-week sentiment.