{"slug": "how-to-catch-llm-regressions-before-a-model-upgrade-ships", "title": "How to catch LLM regressions before a model upgrade ships", "summary": "A new evalshift/llm-regression-example repository demonstrates a CI-gated process for catching LLM regressions before a model upgrade ships, built on six real failures from upgrading a three-route agent from Sonnet 5 to Sonnet 5.5. Replaying 30 recorded production turns on both models, 6 of the 30 failed a check on the new model that the old one passed, including a refusal flipping into a hallucinated tool call, tool argument drift from date=\"2026-09-08\" to date=\"today\", a priority drop from urgent to high, and a billing-to-bug classification shift. The process samples production logs into a golden set, writes deterministic assertions, diffs baseline against candidate pass rates, and gates the diff in CI; the recorded outputs let the diff run offline with no API key.", "body_md": "[← all posts](https://www.evalshift.dev/blog)\n\n# How to catch LLM regressions before a model upgrade ships\n\nSample production logs into a golden set, write deterministic assertions, diff baseline against candidate and gate it in CI. Built on six real failures.\n\nYour provider is retiring the model you run in production, or a cheaper one just shipped and someone has done the arithmetic. Either way, one string in a config file is about to change, and nothing in your test suite has an opinion about what the model says. Here is the process for giving it one: sample real inputs from production into a golden set, write a deterministic assertion for each way an upgrade can break you, run the current model and the candidate over the same cases and diff the pass rates, then put that diff in CI so the next upgrade is a pull request check instead of a week of support tickets.\n\nEverything below runs from one small repo,\n[evalshift/llm-regression-example ↗](https://github.com/evalshift/llm-regression-example), and\nevery failure in it is real: it comes from upgrading a three-route agent from Sonnet 5 to\nSonnet 5.5. The repo ships the recorded outputs, so the diff runs offline, with no API key.\n\nThe reason this needs a process rather than an afternoon in the playground is that the bugs a model upgrade causes are mostly silent. The output still reads fine. A tool argument changed shape, an enum value moved one notch, a ticket landed in a different queue, and nobody sees it until something downstream counts it.\n\n## ## What actually breaks when you switch models\n\nThe agent in the example has three routes: a day planner and a unit converter that call tools, and a ticket triage classifier that returns JSON. Thirty recorded production turns, replayed on both models. Six of the thirty failed a check on the new model that the old one passed, and every one of them produced output a reviewer would have waved through.\n\n| Failure type | What it looked like in our run | How to catch it | \n|---|---|---|\n| Refusal flips into a hallucinated tool call | \"Convert 10 kilometres to kilograms.\" Sonnet 5 explained the units don't convert. Sonnet 5.5 called `convert_units(km → kg)` and said nothing. | Pin cases where production answered without a tool, and assert no tool is called. | \n| Tool argument drift | \"What's on my calendar today?\" went from `get_schedule(date=\"2026-09-08\")` to`get_schedule(date=\"today\")` , against a prompt rule to resolve relative dates. | A regex on the argument: `^\\d{4}-\\d{2}-\\d{2}$` . | \n| Extra tool calls | \"Any notes about Sam's gift, and when is that task due?\" gained a second call, `get_tasks` , on top of`search_notes` . Two cases did this. | Compare the set of tools called against the set production called. | \n| Schema-valid, wrong value | A double-charge ticket dropped from priority `urgent` to`high` . Still valid JSON, still an allowed enum value. | Field equality against the baseline on every enum field. Schema validation alone passes this. | \n| Classification shift | A ticket about app logouts and a late refund moved from `billing` to`bug` . Different queue, different owner. | Same field-equality check. | \n| Format leakage | The old model wrapped its JSON in markdown code fences on 8 of 9 tickets, against an explicit instruction not to. The new one never did. | A strict `json.loads` on the raw output, separate from the lenient parse your app uses. | \n| Verbosity and tone | Suggested replies grew 8% longer and more specific: the new model asks for the Safari version and offers Chrome as a workaround. | Length bounds on fields that have them. A pairwise judge for the rest, advisory only. | \n| Latency and cost | Converter latency rose 47%. Triage got 32% faster and 27% cheaper. Planner barely moved. | Mean latency and cost per route, not per run. A single average would have shown a wash. | \n\nThe table is from a toy agent, but none of the rows is toy-specific. Every production agent I have looked at has an enum field, a tool argument with a required shape, and a case where the right answer is to not call a tool. The upgrade that gets one of those wrong does not throw.\n\n## ## Step 1: build a golden set from production logs\n\nThe playground fails for a structural reason: the prompts you can think of are the prompts you already handle. The cases that break are the ones a user typed, so the golden set has to come from logs. Log one line per model turn, with everything the assertions will later need:\n\n```\n{\"request_id\": \"planner-000\", \"ts\": \"2026-10-12T20:35:51Z\", \"route\": \"planner\",\n \"history\": [], \"input\": \"What's on my calendar today?\",\n \"output\": \"You have Team standup at 9:30–10:00 AM on Zoom, and a Dentist appointment ...\",\n \"tool_calls\": [{\"name\": \"get_schedule\", \"args\": {\"date\": \"2026-09-08\"}}],\n \"latency_ms\": 1863}\n```\n\nThe `tool_calls` and the `output` are what production did, and they become the expectations.\nThen sample. Aim for 100 to 300 lines, stratified by route or feature, because a 5% regression on\na route that is 5% of traffic is invisible in an unstratified sample. Strip PII before anything\ntouches disk. Drop repeated inputs: twenty copies of \"where is my order\" make the suite look twenty\ntimes more certain than it is. And tag the edge cases you already know about, so sampling can\nnever drop them.\n\n``` php\ndef build(lines: list[dict], per_route: int) -> list[dict]:\n    by_route: dict[str, list[dict]] = defaultdict(list)\n    seen: set[tuple[str, str]] = set()\n    for line in sorted(lines, key=lambda x: x[\"ts\"]):\n        key = (line[\"route\"], line[\"input\"].strip().lower())\n        if key in seen:\n            continue\n        seen.add(key)\n        by_route[line[\"route\"]].append(line)\n\n    cases = []\n    for route, route_lines in sorted(by_route.items()):\n        edge = [x for x in route_lines if \"edge\" in x.get(\"tags\", [])]\n        rest = [x for x in route_lines if x not in edge]\n        chosen = edge + rest[: max(per_route - len(edge), 0)]\n        chosen.sort(key=lambda x: x[\"ts\"])\n        for line in chosen:\n            cases.append(\n                {\n                    \"id\": line[\"request_id\"],\n                    \"route\": route,\n                    \"history\": [\n                        {\"role\": h[\"role\"], \"content\": strip_pii(h[\"content\"])}\n                        for h in line.get(\"history\", [])\n                    ],\n                    \"input\": strip_pii(line[\"input\"]),\n                    \"expect\": expectation(line),\n                }\n            )\n    return cases\npython build_golden.py logs/production.jsonl > golden.jsonl\n```\n\n`expectation()` writes the tools production called with their arguments, or, for the JSON route,\nthe values of the enum fields. Review the file before committing it, then freeze it. The\nexpectations say \"do what production did\", and production is a baseline, not an oracle. If you\nare still editing cases while you compare models, you are comparing two moving things.\n\n## ## Step 2: write assertions, cheapest first\n\nHere is the opinionated part. Every regression in the table above was caught by a check that makes no model call, runs in microseconds, and could be explained to a new hire in one sentence. Embedding similarity scored the priority downgrade at 0.95 and the queue change at 0.87. An LLM judge, asked which output was better, preferred the new model on 8 of 9 tickets, which is true and useless: better prose, wrong priority.\n\nSo start with the boring checks, and encode them as data rather than prose:\n\n```\n# Required keys of the triage JSON and their allowed values (None = free text).\nJSON_SCHEMA: dict[str, set[str] | None] = {\n    \"category\": {\"billing\", \"bug\", \"feature_request\", \"account\", \"other\"},\n    \"priority\": {\"low\", \"medium\", \"high\", \"urgent\"},\n    \"sentiment\": {\"negative\", \"neutral\", \"positive\"},\n    \"summary\": None,\n    \"suggested_reply\": None,\n}\nSUMMARY_MAX_WORDS = 20\n\nISO_DATE = re.compile(r\"^\\d{4}-\\d{2}-\\d{2}quot;)\nISO_CURRENCY = {\"USD\", \"EUR\", \"GBP\", \"JPY\", \"CHF\"}\n# What the prompts promise the tools will receive. The model is supposed to\n# normalise before calling; these rules say whether it did.\nARG_RULES = {\n    \"get_schedule\": {\"date\": _is_iso_date},\n    \"days_between\": {\"start_date\": _is_iso_date, \"end_date\": _is_iso_date},\n    \"convert_currency\": {\n        \"from_currency\": lambda v: v in ISO_CURRENCY,\n        \"to_currency\": lambda v: v in ISO_CURRENCY,\n    },\n    \"convert_units\": {\"value\": _is_number},\n}\n\n# Arguments the model phrases freely. A different wording is drift worth\n# printing, not a failure.\nFREE_TEXT_ARGS = {(\"search_notes\", \"query\")}\n```\n\nTwo kinds of check are mixed in here, and it is worth keeping them straight. Baseline checks compare the candidate against what production did: same set of tools, same arguments, same enum values. Contract checks encode what the prompt promises and hold no matter what production did: the JSON parses, the date is ISO, the summary is under twenty words. The baseline checks catch change; the contract checks catch things both models might be getting wrong.\n\nThe strict JSON parse deserves its own line. Your app almost certainly strips fences before parsing, because at some point a model added them and someone fixed it in the parser. Keep that parser. But assert on the raw output too, so a model that starts, or stops, adding fences shows up as a change rather than silently depending on a workaround.\n\nFree text gets a judge, and only free text. The suggested reply in a triage ticket has no deterministic contract beyond its length, so a pairwise judge asking \"which reply is more specific\" is the right tool. Run it advisory, read what it says, and do not let it gate anything until you have audited its calls by hand. Everything above it in this section is the gate.\n\n## ## Step 3: run baseline against candidate and diff\n\nRun every golden case through both models at temperature 0, apply every check to every output, and compare pass rates per route and per check. Not one average: a run where triage improves and the planner regresses can average out to nothing.\n\nNondeterminism is real even at temperature 0. The baseline model disagreed with itself on one of\nnine tickets between the original capture and the replay: `sentiment` flipped from `negative` to\n`neutral` on the same input, same prompt, same model. That is why the runner takes several samples\nper case and scores the fraction that pass, rather than one outcome. Three samples per case is\na reasonable default; the pass rate then has a floor you can reason about.\n\nHere is the real output, trimmed to the rows that matter, with one sample per case because that is what the recorded run had:\n\n``` php\n30 cases, 1 sample(s) each: claude-sonnet-5 -> claude-sonnet-5-5\n\nroute      check             n  baseline  candidate   delta\nconverter  answered          1   100.0%      0.0% -100.0  REGRESSED\nconverter  arg_format        9   100.0%    100.0%   +0.0\nconverter  tool_args         9   100.0%    100.0%   +0.0\nconverter  tool_set         10   100.0%     90.0%  -10.0  REGRESSED\nplanner    arg_format       11   100.0%     90.9%   -9.1  REGRESSED\nplanner    tool_args        11   100.0%     90.9%   -9.1  REGRESSED\nplanner    tool_set         11   100.0%     81.8%  -18.2  REGRESSED\ntriage     fields            9    88.9%     66.7%  -22.2  REGRESSED\ntriage     json_schema       9   100.0%    100.0%   +0.0\ntriage     json_strict       9    11.1%    100.0%  +88.9  improved\ntriage     summary_length    9   100.0%    100.0%   +0.0\n\nCases where claude-sonnet-5-5 fails a check claude-sonnet-5 passed:\n\nconverter-008  'Convert 10 kilometres to kilograms.'\n  tool_set       extra ['convert_units']\n  answered       no text reply\n  claude-sonnet-5:    \"That's not a valid conversion—kilometres measure distance while kilograms measure mass, so there's no meani...\"\n  claude-sonnet-5-5:  convert_units({\"value\": 10, \"from_unit\": \"km\", \"to_unit\": \"kg\"})\n\nplanner-000  \"What's on my calendar today?\"\n  tool_args      get_schedule.date: '2026-09-08' -> 'today'\n  arg_format     get_schedule.date='today'\n  claude-sonnet-5:    get_schedule({\"date\": \"2026-09-08\"})\n  claude-sonnet-5-5:  get_schedule({\"date\": \"today\"})\n\ntriage-003  'I was charged twice for my subscription this month and need the duplicate refunded today — this is unacceptable.'\n  fields         priority: 'urgent' -> 'high'\n  claude-sonnet-5:    '``` json {   \"category\": \"billing\",   \"priority\": \"urgent\",   \"sentiment\": \"negative\",   \"summary\": \"Custome...'\n  claude-sonnet-5-5:  '{\"category\":\"billing\",\"priority\":\"high\",\"sentiment\":\"negative\",\"summary\":\"Customer was charged twice for th...'\n\nroute      metric       claude-sonnet-5 claude-sonnet-5-5   change\nconverter  latency_ms           1568 ms          2312 ms    +47%\nconverter  cost_usd             $0.0057          $0.0057     -1%\nplanner    latency_ms           1481 ms          1538 ms     +4%\ntriage     latency_ms           3035 ms          2071 ms    -32%\ntriage     cost_usd             $0.0028          $0.0020    -27%\n\nFAIL: 6 check(s) dropped more than 2% (worst: converter/answered -100.0 points)\n```\n\nThe threshold is a policy choice, and it has to respect the denominator. With eleven planner cases, one failure is a nine-point drop, so a two-point threshold means zero tolerance for that route; that is fine as long as you know it. With 300 cases the same threshold tolerates six. Set it per route if the routes differ in how much they matter, and write it down before you look at the numbers, because a threshold chosen after seeing the diff is a negotiation, not a gate.\n\nLook at the `n` column too. The `answered` check ran on one case, because production answered\nwithout a tool exactly once in thirty turns. That is a hole in the golden set, not a finding about\nthe model, and it is the kind of thing this table is for: it tells you which rows you would act\non and which rows you cannot yet trust.\n\n## ## Step 4: gate it in CI\n\nThe run that changes outcomes is the one attached to the pull request that bumps the model string. Trigger it on the files that can move the numbers, and nothing else, because every run spends real money:\n\n```\nname: llm-regression\n\n# Run only when something that can move the numbers changes: the prompts and\n# model names (agent.py), the assertions, the golden set, or the runner.\non:\n  pull_request:\n    paths:\n      - agent.py\n      - checks.py\n      - golden.jsonl\n      - run_eval.py\n      - .github/workflows/llm-regression.yml\n  workflow_dispatch:\n\npermissions:\n  contents: read\n\njobs:\n  eval:\n    runs-on: ubuntu-latest\n    env:\n      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}\n    steps:\n      - uses: actions/checkout@v7\n      - uses: actions/setup-python@v6\n        with:\n          python-version: \"3.12\"\n      - run: pip install -e .\n\n      # The baseline is the model production runs today. Its outputs only\n      # change when the prompt or the golden set changes, so key the cache on\n      # those and the candidate is the only side that gets re-run per PR.\n      # (This repo commits results/ so it runs offline; in your own repo add\n      # results/ to .gitignore and let this cache carry the baseline.)\n      - name: Restore cached baseline results\n        uses: actions/cache@v4\n        with:\n          path: results/claude-sonnet-5.jsonl\n          key: baseline-claude-sonnet-5-${{ hashFiles('golden.jsonl', 'agent.py') }}\n\n      - name: Baseline vs candidate\n        run: python run_eval.py --repeats 1 --max-drop 0.02\n```\n\nThe cache is the part people skip. The baseline's outputs only change when the prompt or the golden set changes, so key the cache on those files and the candidate becomes the only side that costs money per pull request. The runner reads a model's results from disk when they exist and only calls the model for what is missing, which is also what lets the example repo run with no key at all.\n\nShip it in report-only mode for the first week. Let it comment, read what it says, and turn on the failure exit once you agree with it. A gate people learn to re-run until it passes is worse than no gate.\n\n## ## Step 5: triage what failed\n\nA red check is a list of decisions, and each one has four possible answers: fix the prompt, add an output parser, pin the old model for that route, or accept the change and update the golden set. Walking the six from our run:\n\n- +`get_schedule(date=\"today\")` : fix the prompt. The rule was there and the new model weighs it\ndifferently; it needs to be louder, and the tool should reject non-ISO dates so the next drift\nfails loudly. The tool in the original app happened to accept \"today\", which is exactly why the\noutput looked fine.\n- +The hallucinated `convert_units` call: fix the prompt with an explicit \"if the request is not a\nvalid conversion, say so without calling a tool\", and keep the case pinned. One such case in\nthirty is too few; add more impossible requests to the golden set.\n- +The two extra `get_tasks` calls: accept. Both questions genuinely span notes and tasks, and the\nnew model is reading them more carefully than the old one did. Update the expected tool set and\nmove on.\n- +The fences: accept, and keep the strict check. The new model obeys the instruction. The parser stays because the next model may not.\n- +`billing` to`bug` on the logout-plus-refund ticket: a judgment call about which queue owns a\nmixed ticket. Decide, write the rule into the prompt, and the check will enforce it from now on.\n\nThe priority downgrade is the one worth a story. The ticket said \"charged twice\", \"refunded today\" and \"unacceptable\". The prompt defines urgent as \"the customer is blocked or losing money right now\", and whether a duplicate charge counts is a question the prompt never answered, so both models' calls are defensible. The decision took two minutes: a disputed charge is urgent, say so in the prompt, keep the field check. Without the diff, that two-minute decision would have been made by the model, silently, for every double-charge ticket after the upgrade, and the first sign would have been an SLA report a month later.\n\n## ## Doing this with EvalShift\n\nDisclosure: I'm the founder of EvalShift, and the three scripts above are the method it\nautomates. The capture SDK replaces the log exporter, `evalshift.yaml` replaces `checks.py`, and\nthe run comes back with paired statistics and a policy verdict instead of a pass-rate threshold.\n\n``` python\nfrom evalshift import capture\n\n@capture.agent(suite=\"triage\", redact=True)   # no-op unless EVALSHIFT_CAPTURE=1\ndef handle_ticket(message): ...\nphp\npip install evalshift\nevalshift capture sync      # recorded runs -> golden suite\nevalshift compare           # baseline vs candidate -> report.html + verdict\n- uses: evalshift/evalshift-action@v0\n  with:\n    token: ${{ secrets.EVALSHIFT_TOKEN }}\n    fail-on: policy\n```\n\nThe checks in this post map onto its evaluators one to one: `tool_selection` is the `tool_set`\ncheck, `tool_arguments` is `tool_args`, and the `ARG_RULES` table is a `trace_invariants` block\nin YAML, with the same ISO-date regex. The example repo ships that file too. What you get on top is\nthe part that is tedious to write yourself: Wilson intervals on every rate so a small suite returns\ninconclusive instead of a false fail, per-slice budgets, and a single-file report with the\nper-example diff. The reports behind this post were EvalShift runs; the scripts reproduce them\nfrom the recorded outputs. Docs: [/docs/golden-suite](https://www.evalshift.dev/docs/golden-suite),\n[/docs/evaluators](https://www.evalshift.dev/docs/evaluators), [/docs/action](https://www.evalshift.dev/docs/action).\n\n## ## FAQ\n\n**How many test cases do I need?** Enough that every route you would act on has about twenty,\nafter de-duplication. Thirty total was enough to show every failure type here and not enough to\nmake a decision on any single route. One hundred to three hundred, stratified by route, is the\nrange where pass-rate deltas start meaning something.\n\n**Can I use an LLM as a judge?** For free-text fields, yes, and only after the deterministic\nchecks have had their turn. In our run the judge preferred the new model on 8 of 9 tickets while\nthe priority field regressed. Keep judge results advisory until you have audited the pairs it\ngot wrong.\n\n**How do I handle nondeterministic outputs?** Temperature 0, several samples per case, and pass\nfractions instead of single outcomes. The baseline model here changed a sentiment label between\ntwo runs of the same input. One sample per case cannot distinguish that from a regression; three\ncan.\n\n**How often should I re-run the tests?** On every change to the prompt, the model string, the\ntool definitions or the golden set, which is what the workflow's `paths` filter encodes. Not on\nevery commit: the run costs money and nothing else can move the numbers.\n\n**Should I change the prompt in the same pull request as the model?** No. Run the new model on the\nold prompt first so the diff isolates the model. Then tune the prompt in its own pull request\nagainst the same frozen cases. Two variables in one comparison leave nobody able to say which one\nmoved the score.\n\n## ## Keep reading\n\n- +[What actually breaks when you switch LLMs](https://www.evalshift.dev/blog/what-breaks-when-you-switch-llms) — the\nlonger argument for why the worst regressions read as correct.\n- +[Build a golden eval suite from production traffic](https://www.evalshift.dev/blog/build-a-golden-suite-from-production-traffic) — the capture-first version of step one.\n- +[LLM regression testing in CI](https://www.evalshift.dev/blog/llm-regression-testing-in-ci) — the pull request gate with\nthe GitHub Action, including the calibration week.\n- +[When to trust an LLM judge](https://www.evalshift.dev/blog/when-to-trust-an-llm-judge) — what pairwise judging is good\nat, and where it quietly misleads you.", "url": "https://wpnews.pro/news/how-to-catch-llm-regressions-before-a-model-upgrade-ships", "canonical_source": "https://www.evalshift.dev/blog/llm-regression-testing-model-upgrade", "published_at": "2026-10-12 00:00:00+00:00", "updated_at": "2026-10-12 21:20:38.054497+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "mlops", "developer-tools", "ai-agents"], "entities": ["evalshift", "llm-regression-example", "Sonnet 5", "Sonnet 5.5"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-catch-llm-regressions-before-a-model-upgrade-ships", "markdown": "https://wpnews.pro/news/how-to-catch-llm-regressions-before-a-model-upgrade-ships.md", "text": "https://wpnews.pro/news/how-to-catch-llm-regressions-before-a-model-upgrade-ships.txt", "jsonld": "https://wpnews.pro/news/how-to-catch-llm-regressions-before-a-model-upgrade-ships.jsonld"}}