{"slug": "cheap-model-first-strong-model-on-failure-building-an-auditable-two-tier-llm", "title": "Cheap Model First, Strong Model on Failure: Building an Auditable Two-Tier LLM Pipeline", "summary": "A developer has built an auditable two-tier LLM pipeline that routes work to a cheap model by default and escalates to a stronger model only when an external deterministic check fails. The implementation, written in Python using only the standard library and requests, logs every routing decision to a JSONL audit file. The design explicitly avoids using an LLM as a judge, relying instead on test suites, JSON schema validators, or regex patterns for validation.", "body_md": "There's a pattern I see every release cycle: a new budget-friendly model ships, the discourse explodes with hot takes, and within 48 hours half my feed has declared it a drop-in replacement for everything. The claim might even be true. But here's what nobody posting those takes can tell you: whether it's true *for your specific workload*. And most of the time, that's the only question that matters.\n\nA healthier mental model: stop treating model selection as a one-time shopping decision and start treating it as a runtime policy. Route work to the inexpensive option by default, check the output with something that isn't a model, and only pay for the heavyweight option when the check fails. Below is a working implementation of that policy, plus the measurement discipline that turns it from a hunch into an auditable system.\n\nMy configuration is deliberately boring:\n\nOn the cost side, one note: I iterate on this pipeline using MonkeyCode, which at the time of writing provides free model access along with a free server option, so experimentation doesn't rack up a bill. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The pipeline itself is provider-agnostic — anything speaking the OpenAI-compatible chat API drops in, so treat endpoints as configuration, not commitment.\n\nThe single most important design decision: never let a model decide whether its own output (or a peer's) was good enough. LLMs report confidence regardless of correctness. Instead, escalate only when an *external, deterministic check* fails — a test suite, a parser, a schema validator, a diff against expected output.\n\nHere's a compact implementation. Standard library plus `requests`\n\n, fully rerunnable, and every routing decision gets written to a JSONL audit log:\n\n``` python\n# lane_router.py\nimport hashlib, json, subprocess, time\nfrom dataclasses import dataclass, asdict\n\nimport requests\n\n@dataclass\nclass RouteRecord:\n    job_id: str\n    lane: str            # \"A\" or \"B\"\n    fell_back: bool\n    check_ok: bool\n    seconds: float\n    prompt_fingerprint: str\n\ndef chat(endpoint: str, model: str, prompt: str) -> str:\n    resp = requests.post(\n        f\"{endpoint}/v1/chat/completions\",\n        json={\n            \"model\": model,\n            \"messages\": [{\"role\": \"user\", \"content\": prompt}],\n            \"temperature\": 0.2,\n        },\n        timeout=120,\n    )\n    resp.raise_for_status()\n    return resp.json()[\"choices\"][0][\"message\"][\"content\"]\n\ndef objective_check(job: dict, candidate: str) -> bool:\n    \"\"\"Deterministic validation only. No LLM-as-judge allowed here.\"\"\"\n    mode = job[\"check\"]\n    if mode == \"pytest\":\n        path = job[\"write_to\"]\n        with open(path, \"w\") as fh:\n            fh.write(strip_fences(candidate))\n        run = subprocess.run(job[\"command\"], shell=True,\n                             capture_output=True, timeout=90)\n        return run.returncode == 0\n    if mode == \"valid_json\":\n        try:\n            parsed = json.loads(strip_fences(candidate))\n        except json.JSONDecodeError:\n            return False\n        required = job.get(\"required_keys\", [])\n        return all(k in parsed for k in required)\n    if mode == \"regex\":\n        import re\n        return re.fullmatch(job[\"pattern\"], candidate.strip()) is not None\n    raise ValueError(f\"unsupported check: {mode}\")\n\ndef strip_fences(text: str) -> str:\n    \"\"\"Pull code out of a markdown fence if present; else return as-is.\"\"\"\n    if \"```\n\n\" not in text:\n        return text\n    block = text.split(\"\n\n```\")[1]\n    lines = block.splitlines()\n    if lines and lines[0].strip().isalpha():  # language tag line\n        lines = lines[1:]\n    return \"\\n\".join(lines)\n\ndef route(job: dict, lanes: list[dict], log_path: str = \"routes.jsonl\") -> RouteRecord:\n    fp = hashlib.sha256(job[\"prompt\"].encode()).hexdigest()[:12]\n    fell_back = False\n    for idx, lane in enumerate(lanes):\n        start = time.time()\n        candidate = chat(lane[\"endpoint\"], lane[\"model\"], job[\"prompt\"])\n        elapsed = round(time.time() - start, 2)\n        passed = objective_check(job, candidate)\n        if passed or idx == len(lanes) - 1:\n            record = RouteRecord(\n                job_id=job[\"id\"],\n                lane=lane[\"label\"],\n                fell_back=fell_back,\n                check_ok=passed,\n                seconds=elapsed,\n                prompt_fingerprint=fp,\n            )\n            with open(log_path, \"a\") as fh:\n                fh.write(json.dumps(asdict(record)) + \"\\n\")\n            return record\n        fell_back = True\n```\n\nWiring it up:\n\n```\nlanes = [\n    {\"label\": \"A\", \"endpoint\": \"https://lane-a-endpoint\", \"model\": \"current-budget-model\"},\n    {\"label\": \"B\", \"endpoint\": \"https://lane-b-endpoint\", \"model\": \"premium-model\"},\n]\n\njob = {\n    \"id\": \"csv-to-json-migration-014\",\n    \"prompt\": (\n        \"Convert the transformation in migrate.py so it emits newline-delimited JSON. \"\n        \"Return only the complete updated file inside a code fence.\"\n    ),\n    \"check\": \"pytest\",\n    \"write_to\": \"migrate.py\",\n    \"command\": \"python -m pytest tests/test_migrate.py -q\",\n}\n\nprint(route(job, lanes))\n```\n\nThe router code is maybe a weekend of effort. The compounding value lives in `routes.jsonl`\n\n. After a few weeks of real traffic you can compute things that are otherwise pure speculation:\n\n`fell_back`\n\n. If data-formatting jobs pass on Lane A 92% of the time, Lane A is a rational default there. If multi-file refactors fail 60% of the time, Lane A is a false economy for that family — you're paying for a doomed first attempt plus added latency on most calls.Operating rules I'd insist on:\n\nIf most of your workload is unverifiable generation, or everything you run is on a hard latency budget, honestly — skip the router. Picking the strong model outright is the simpler and more correct engineering decision in that world.\n\nPull your last ~50 real prompts, sort them into \"has a deterministic check\" versus \"doesn't,\" and push the checkable subset through the two-lane setup for a week. If you want the measurement phase to cost nothing, MonkeyCode's free model access and free server option work fine as Lane A and host while you gather data — and since the log format is provider-neutral, whatever you learn transfers when you point the lanes elsewhere.\n\nThe question \"is the cheap model good enough?\" has an answer, and it's already sitting in your prompt history. Measure it; don't outsource the decision to launch-week sentiment.", "url": "https://wpnews.pro/news/cheap-model-first-strong-model-on-failure-building-an-auditable-two-tier-llm", "canonical_source": "https://dev.to/codego_3211/cheap-model-first-strong-model-on-failure-building-an-auditable-two-tier-llm-pipeline-32c", "published_at": "2026-08-13 04:00:12+00:00", "updated_at": "2026-08-13 04:21:49.606892+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["MonkeyCode"], "alternates": {"html": "https://wpnews.pro/news/cheap-model-first-strong-model-on-failure-building-an-auditable-two-tier-llm", "markdown": "https://wpnews.pro/news/cheap-model-first-strong-model-on-failure-building-an-auditable-two-tier-llm.md", "text": "https://wpnews.pro/news/cheap-model-first-strong-model-on-failure-building-an-auditable-two-tier-llm.txt", "jsonld": "https://wpnews.pro/news/cheap-model-first-strong-model-on-failure-building-an-auditable-two-tier-llm.jsonld"}}