How to test an LLM model migration before you ship it

A new open-source tool, Evalshift, provides a repeatable method for testing large language model (LLM) migrations by freezing a golden suite of recorded agent runs, running both source and target models over identical cases, and analyzing paired diffs across output quality, tool-call behavior, and cost/latency. The tool, available as PyPI packages `evalshift` and `evalshift-sdk`, addresses the gap where CI cannot validate model output changes, helping teams prove a model swap is safe before shipping.

← all posts /blog How to test an LLM model migration before you ship it A repeatable method for proving a model swap is safe: freeze a golden suite, run both models paired, and read the diff before your users do. Swapping the model behind a production feature is a code change, except nothing checks it. The provider ships a newer version, you edit one string in a config file, and CI stays green because CI never had an opinion about model output. Whatever broke shows up later as support tickets, by which point the deploy that caused it is twenty deploys back. The usual substitute for evidence is the playground: paste in twenty prompts, read the answers, decide it looks fine. That fails for a structural reason, not a diligence one. The prompts you can think of are the prompts you already handle well — the eleven-turn conversation where the model drops a constraint set in turn three, the refund request where the new model calls issue refund before lookup order , the input in a language your template never anticipated. You cannot type those from memory. You have to have recorded them. What "safe" actually means here Safe does not mean "the new model is better." It means you can state what changed, in which direction, on which cases, and with what confidence — precisely enough that a colleague who disagrees has to argue with a number instead of your intuition. That splits into three failure classes, independent enough that each needs its own instrumentation: - +Output quality drift: the answer is still fluent and on topic, but less correct, less complete, or no longer the shape your downstream code parses. - +Tool-call behavior drift: the agent picks a different tool, calls tools in a different order, skips a verification step, or passes subtly different arguments. The output text can look identical while the trace underneath has changed. - +Cost and latency drift: the same answers, slower, or at three times the tokens per turn. A migration can pass one class and fail another. A cheaper model that answers just as well but issues one extra tool call per turn is not a cost reduction, and reading outputs will never tell you that. Step 1 — freeze a golden suite The suite has to be fixed before you touch models. If you are still editing cases while you compare, you are comparing two moving things, and the diff between them tells you nothing about either. Real traffic beats invented prompts, for the same reason the playground fails. The capture SDK records agent runs in process to .evalshift/captures/ , and evalshift capture sync promotes every capture into .evalshift/suites/<suite /golden.jsonl — one SuiteExample per conversation turn, grouped by conversation id and ordered by turn index . The case that broke in turn seven stays a case about turn seven. The CLI evalshift and the capture SDK evalshift-sdk are separate PyPI packages that share the top-level import name evalshift . Install them in separate virtual environments. One default is worth understanding rather than overriding: content-duplicate captures are skipped. That is not housekeeping. Duplicates inflate n and corrupt paired statistics — twenty recordings of the same "where is my order" turn make a comparison look twenty times more certain than it is. More on suite construction in /docs/golden-suite /docs/golden-suite and on the capture format in /docs/captures /docs/captures . Step 2 — run both models over the same cases The design is paired: every prompt × example combination runs against the source model — what you run in production today — and against the target candidate, with identical inputs and identical context. Pairing is what makes the arithmetic honest. You subtract per example, so the fact that some cases are inherently harder than others cancels instead of swamping the signal. evalshift all --from gemini-3.1-flash --to gemini-3.1-pro --suite-name checkout-agent --from and --to override defaults.source model and defaults.target model from your config, so the file records the migration you are planning while the flags let you audition candidates without editing it. evalshift all chains the whole pipeline: doctor → run → evaluate → analyze → report. Rehearse for free first. evalshift demo scaffolds a runnable project, and evalshift all --offline --yes --open replays canned fixtures through the same pipeline with no API keys and no spend. Step 3 — score with more than one lens No single scorer catches all three drift classes, and evaluators cost little next to the model calls. Configure several. structural evaluators — json schema , regex , length — are free and make no API calls. Being deterministic makes them the cheapest possible alarm. If your output has any contract, encode it here: a schema that stops validating needs no judgment call. semantic compares embeddings. The source output is pinned at 1.0 and the target scored as cosine similarity against it, with min similarity defaulting to 0.9 . That answers "did the meaning move," not "is it better" — useful as a drift alarm, misleading as a quality score. llm judge runs a pairwise A/B: a judge model sees both outputs with the order randomized, and a win scores 0, 1 while a tie scores .5, .5 . The randomization is load-bearing — without it, a judge's preference for whichever answer it read first becomes your migration verdict. tool selection and tool arguments cover agents. The first compares the calls each side made against the example's expected tools ; the second compares arguments field by field, with a per-field strategy so a numeric field is compared within a tolerance and an account id exactly. Every evaluator config also takes blocking: bool = true . Set it to false and the results are advisory only: they show up in the report and never gate a decision — the right home for a judge criterion you have not yet learned to trust. Full reference: /docs/evaluators /docs/evaluators . Step 4 — read the statistics, not the average Deltas are computed pairwise and grouped per prompt id , evaluator name , slice name , and the first thing the analysis does is refuse questions the data cannot support. Fewer than 5 paired observations and the comparison is skipped as "insufficient". Between 5 and 20 it is tested but flagged uncertain. The test is chosen rather than assumed: Shapiro-Wilk at α=0.05 on the deltas picks a paired t-test when they look normal and a Wilcoxon signed-rank test when they do not. Every testable comparison in the run then goes through a Benjamini-Hochberg FDR correction at α=0.05, because a suite with forty comparisons will hand you two "significant" findings by luck alone if nobody corrects for it. Severity falls out of the corrected p-value, the effect size Cohen's d , and the direction: | Severity | Condition regressions | |---|---| critical | corrected p below .01 and effect size above 0.8 | high | significant, effect size above 0.5 | medium | significant, effect size above 0.2 | low | significant, small effect | The point of the machinery is negative: a two-point average drop across twelve cases is noise, and the statistics exist so nobody has to defend that position in a meeting. The method is written up in /docs/methodology /docs/methodology . Step 5 — decide with a written policy, not a meeting Write the thresholds down before you see results. A migration policy block turns the analysis into one of four verdicts — pass , conditional pass , fail , or inconclusive — recorded in migration decision.json . Only blocking evaluators gate it; advisory results are summarized separately and never flip the verdict. The interesting verdict is inconclusive . Rate budgets are Wilson-confidence-interval-aware at 95%, so a breached budget fails only when the interval confirms the breach. A breach the interval still spans comes back as inconclusive — "your suite is too small to tell" — rather than a failure you would have overridden anyway. Count, cost and latency budgets are exact and always conclusive. A fail means a conclusive budget failure or a blocking critical or high comparison. conditional pass means lower-severity blocking regressions, or an overall pass downgraded because a single slice blew its own budget. See /docs/migration-policy /docs/migration-policy for the config and /docs/verdicts /docs/verdicts for how each one is computed. What this looks like in one afternoon - +Instrument your agent with the capture SDK and record a day of real traffic. - +Run evalshift capture sync to turn those captures into a golden suite. - +Run the pipeline offline once, to validate the suite before it costs anything. - +Run it live against both models, paired. - +Read the report — start at the severities, not the averages. - +Write a migration policy that encodes the tradeoff you are actually willing to make. - +Wire the same run into CI so the next model bump is a pull request check instead of an afternoon. Steps one through six are a one-time cost. The seventh is what keeps them from recurring every quarter. Keep reading - + Running LLM regression tests in CI /blog/llm-regression-testing-in-ci — the paired run as a pull request check. - + When to trust an LLM judge /blog/when-to-trust-an-llm-judge — what pairwise judging is good at, and where it quietly misleads you. - + Getting started /docs/getting-started — install, scaffold, and a first offline run.