{"slug": "how-to-test-an-llm-model-migration-before-you-ship-it", "title": "How to test an LLM model migration before you ship it", "summary": "A new open-source tool, Evalshift, provides a repeatable method for testing large language model (LLM) migrations by freezing a golden suite of recorded agent runs, running both source and target models over identical cases, and analyzing paired diffs across output quality, tool-call behavior, and cost/latency. The tool, available as PyPI packages `evalshift` and `evalshift-sdk`, addresses the gap where CI cannot validate model output changes, helping teams prove a model swap is safe before shipping.", "body_md": "[← all posts](/blog)\n\n# How to test an LLM model migration before you ship it\n\nA repeatable method for proving a model swap is safe: freeze a golden suite, run both models paired, and read the diff before your users do.\n\nSwapping the model behind a production feature is a code change, except nothing checks it. The provider ships a newer version, you edit one string in a config file, and CI stays green because CI never had an opinion about model output. Whatever broke shows up later as support tickets, by which point the deploy that caused it is twenty deploys back.\n\nThe usual substitute for evidence is the playground: paste in twenty prompts, read the answers,\ndecide it looks fine. That fails for a structural reason, not a diligence one. The prompts you can\nthink of are the prompts you already handle well — the eleven-turn conversation where the model\ndrops a constraint set in turn three, the refund request where the new model calls `issue_refund`\n\nbefore `lookup_order`\n\n, the input in a language your template never anticipated. You cannot type\nthose from memory. You have to have recorded them.\n\n## ## What \"safe\" actually means here\n\nSafe does not mean \"the new model is better.\" It means you can state what changed, in which direction, on which cases, and with what confidence — precisely enough that a colleague who disagrees has to argue with a number instead of your intuition.\n\nThat splits into three failure classes, independent enough that each needs its own instrumentation:\n\n- +Output quality drift: the answer is still fluent and on topic, but less correct, less complete, or no longer the shape your downstream code parses.\n- +Tool-call behavior drift: the agent picks a different tool, calls tools in a different order, skips a verification step, or passes subtly different arguments. The output text can look identical while the trace underneath has changed.\n- +Cost and latency drift: the same answers, slower, or at three times the tokens per turn.\n\nA migration can pass one class and fail another. A cheaper model that answers just as well but issues one extra tool call per turn is not a cost reduction, and reading outputs will never tell you that.\n\n## ## Step 1 — freeze a golden suite\n\nThe suite has to be fixed before you touch models. If you are still editing cases while you compare, you are comparing two moving things, and the diff between them tells you nothing about either.\n\nReal traffic beats invented prompts, for the same reason the playground fails. The capture SDK\nrecords agent runs in process to `.evalshift/captures/`\n\n, and `evalshift capture sync`\n\npromotes every\ncapture into `.evalshift/suites/<suite>/golden.jsonl`\n\n— one `SuiteExample`\n\nper conversation turn,\ngrouped by `conversation_id`\n\nand ordered by `turn_index`\n\n. The case that broke in turn seven stays a\ncase about turn seven.\n\nThe CLI (`evalshift`\n\n) and the capture SDK (`evalshift-sdk`\n\n) are separate PyPI packages that share\nthe top-level import name `evalshift`\n\n. Install them in separate virtual environments.\n\nOne default is worth understanding rather than overriding: content-duplicate captures are skipped.\nThat is not housekeeping. Duplicates inflate `n`\n\nand corrupt paired statistics — twenty recordings\nof the same \"where is my order\" turn make a comparison look twenty times more certain than it is.\n\nMore on suite construction in [/docs/golden-suite](/docs/golden-suite) and on the capture format in\n[/docs/captures](/docs/captures).\n\n## ## Step 2 — run both models over the same cases\n\nThe design is paired: every (prompt × example) combination runs against the source model — what you run in production today — and against the target candidate, with identical inputs and identical context. Pairing is what makes the arithmetic honest. You subtract per example, so the fact that some cases are inherently harder than others cancels instead of swamping the signal.\n\n```\nevalshift all --from gemini-3.1-flash --to gemini-3.1-pro --suite-name checkout-agent\n```\n\n`--from`\n\nand `--to`\n\noverride `defaults.source_model`\n\nand `defaults.target_model`\n\nfrom your config,\nso the file records the migration you are planning while the flags let you audition candidates\nwithout editing it. `evalshift all`\n\nchains the whole pipeline: doctor → run → evaluate → analyze →\nreport.\n\nRehearse for free first. `evalshift demo`\n\nscaffolds a runnable project, and\n`evalshift all --offline --yes --open`\n\nreplays canned fixtures through the same pipeline with no API\nkeys and no spend.\n\n## ## Step 3 — score with more than one lens\n\nNo single scorer catches all three drift classes, and evaluators cost little next to the model calls. Configure several.\n\n`structural`\n\nevaluators — `json_schema`\n\n, `regex`\n\n, `length`\n\n— are free and make no API calls.\nBeing deterministic makes them the cheapest possible alarm. If your output has any contract, encode\nit here: a schema that stops validating needs no judgment call.\n\n`semantic`\n\ncompares embeddings. The source output is pinned at 1.0 and the target scored as cosine\nsimilarity against it, with `min_similarity`\n\ndefaulting to `0.9`\n\n. That answers \"did the meaning\nmove,\" not \"is it better\" — useful as a drift alarm, misleading as a quality score.\n\n`llm_judge`\n\nruns a pairwise A/B: a judge model sees both outputs with the order randomized, and a\nwin scores (0, 1) while a tie scores (.5, .5). The randomization is load-bearing — without it, a\njudge's preference for whichever answer it read first becomes your migration verdict.\n\n`tool_selection`\n\nand `tool_arguments`\n\ncover agents. The first compares the calls each side made\nagainst the example's `expected_tools`\n\n; the second compares arguments field by field, with a\nper-field strategy so a numeric field is compared within a tolerance and an account id exactly.\n\nEvery evaluator config also takes `blocking: bool = true`\n\n. Set it to `false`\n\nand the results are\nadvisory only: they show up in the report and never gate a decision — the right home for a judge\ncriterion you have not yet learned to trust. Full reference: [/docs/evaluators](/docs/evaluators).\n\n## ## Step 4 — read the statistics, not the average\n\nDeltas are computed pairwise and grouped per (`prompt_id`\n\n, `evaluator_name`\n\n, `slice_name`\n\n), and the\nfirst thing the analysis does is refuse questions the data cannot support. Fewer than 5 paired\nobservations and the comparison is skipped as \"insufficient\". Between 5 and 20 it is tested but\nflagged uncertain.\n\nThe test is chosen rather than assumed: Shapiro-Wilk at α=0.05 on the deltas picks a paired t-test when they look normal and a Wilcoxon signed-rank test when they do not. Every testable comparison in the run then goes through a Benjamini-Hochberg FDR correction at α=0.05, because a suite with forty comparisons will hand you two \"significant\" findings by luck alone if nobody corrects for it.\n\nSeverity falls out of the corrected p-value, the effect size (Cohen's d), and the direction:\n\n| Severity | Condition (regressions) |\n|---|---|\n`critical` | corrected p below .01 and effect size above 0.8 |\n`high` | significant, effect size above 0.5 |\n`medium` | significant, effect size above 0.2 |\n`low` | significant, small effect |\n\nThe point of the machinery is negative: a two-point average drop across twelve cases is noise, and\nthe statistics exist so nobody has to defend that position in a meeting. The method is written up in\n[/docs/methodology](/docs/methodology).\n\n## ## Step 5 — decide with a written policy, not a meeting\n\nWrite the thresholds down before you see results. A `migration_policy`\n\nblock turns the analysis into\none of four verdicts — `pass`\n\n, `conditional_pass`\n\n, `fail`\n\n, or `inconclusive`\n\n— recorded in\n`migration_decision.json`\n\n. Only blocking evaluators gate it; advisory results are summarized\nseparately and never flip the verdict.\n\nThe interesting verdict is `inconclusive`\n\n. Rate budgets are Wilson-confidence-interval-aware at 95%,\nso a breached budget fails only when the interval confirms the breach. A breach the interval still\nspans comes back as `inconclusive`\n\n— \"your suite is too small to tell\" — rather than a failure you\nwould have overridden anyway. Count, cost and latency budgets are exact and always conclusive.\n\nA `fail`\n\nmeans a conclusive budget failure or a blocking critical or high comparison.\n`conditional_pass`\n\nmeans lower-severity blocking regressions, or an overall pass downgraded because\na single slice blew its own budget. See [/docs/migration-policy](/docs/migration-policy) for the\nconfig and [/docs/verdicts](/docs/verdicts) for how each one is computed.\n\n## ## What this looks like in one afternoon\n\n- +Instrument your agent with the capture SDK and record a day of real traffic.\n- +Run\n`evalshift capture sync`\n\nto turn those captures into a golden suite. - +Run the pipeline offline once, to validate the suite before it costs anything.\n- +Run it live against both models, paired.\n- +Read the report — start at the severities, not the averages.\n- +Write a\n`migration_policy`\n\nthat encodes the tradeoff you are actually willing to make. - +Wire the same run into CI so the next model bump is a pull request check instead of an afternoon.\n\nSteps one through six are a one-time cost. The seventh is what keeps them from recurring every quarter.\n\n## ## Keep reading\n\n- +\n[Running LLM regression tests in CI](/blog/llm-regression-testing-in-ci)— the paired run as a pull request check. - +\n[When to trust an LLM judge](/blog/when-to-trust-an-llm-judge)— what pairwise judging is good at, and where it quietly misleads you. - +\n[Getting started](/docs/getting-started)— install, scaffold, and a first offline run.", "url": "https://wpnews.pro/news/how-to-test-an-llm-model-migration-before-you-ship-it", "canonical_source": "https://www.evalshift.dev/blog/test-llm-model-migration-before-you-ship", "published_at": "2026-07-31 00:00:00+00:00", "updated_at": "2026-08-20 17:14:53.059335+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "mlops", "ai-agents"], "entities": ["Evalshift", "evalshift-sdk", "PyPI", "Gemini 3.1 Flash", "Gemini 3.1 Pro"], "alternates": {"html": "https://wpnews.pro/news/how-to-test-an-llm-model-migration-before-you-ship-it", "markdown": "https://wpnews.pro/news/how-to-test-an-llm-model-migration-before-you-ship-it.md", "text": "https://wpnews.pro/news/how-to-test-an-llm-model-migration-before-you-ship-it.txt", "jsonld": "https://wpnews.pro/news/how-to-test-an-llm-model-migration-before-you-ship-it.jsonld"}}