A repeatable method for proving a model swap is safe: freeze a golden suite, run both models paired, and read the diff before your users do.
Swapping the model behind a production feature is a code change, except nothing checks it. The provider ships a newer version, you edit one string in a config file, and CI stays green because CI never had an opinion about model output. Whatever broke shows up later as support tickets, by which point the deploy that caused it is twenty deploys back.
The usual substitute for evidence is the playground: paste in twenty prompts, read the answers,
decide it looks fine. That fails for a structural reason, not a diligence one. The prompts you can
think of are the prompts you already handle well β the eleven-turn conversation where the model
drops a constraint set in turn three, the refund request where the new model calls issue_refund
before lookup_order
, the input in a language your template never anticipated. You cannot type those from memory. You have to have recorded them.
## What "safe" actually means here #
Safe does not mean "the new model is better." It means you can state what changed, in which direction, on which cases, and with what confidence β precisely enough that a colleague who disagrees has to argue with a number instead of your intuition.
That splits into three failure classes, independent enough that each needs its own instrumentation:
- +Output quality drift: the answer is still fluent and on topic, but less correct, less complete, or no longer the shape your downstream code parses.
- +Tool-call behavior drift: the agent picks a different tool, calls tools in a different order, skips a verification step, or passes subtly different arguments. The output text can look identical while the trace underneath has changed.
- +Cost and latency drift: the same answers, slower, or at three times the tokens per turn.
A migration can pass one class and fail another. A cheaper model that answers just as well but issues one extra tool call per turn is not a cost reduction, and reading outputs will never tell you that.
## Step 1 β freeze a golden suite #
The suite has to be fixed before you touch models. If you are still editing cases while you compare, you are comparing two moving things, and the diff between them tells you nothing about either.
Real traffic beats invented prompts, for the same reason the playground fails. The capture SDK
records agent runs in process to .evalshift/captures/
, and evalshift capture sync
promotes every
capture into .evalshift/suites/<suite>/golden.jsonl
β one SuiteExample
per conversation turn,
grouped by conversation_id
and ordered by turn_index
. The case that broke in turn seven stays a case about turn seven.
The CLI (evalshift
) and the capture SDK (evalshift-sdk
) are separate PyPI packages that share
the top-level import name evalshift
. Install them in separate virtual environments.
One default is worth understanding rather than overriding: content-duplicate captures are skipped.
That is not housekeeping. Duplicates inflate n
and corrupt paired statistics β twenty recordings of the same "where is my order" turn make a comparison look twenty times more certain than it is.
More on suite construction in /docs/golden-suite and on the capture format in /docs/captures.
## Step 2 β run both models over the same cases #
The design is paired: every (prompt Γ example) combination runs against the source model β what you run in production today β and against the target candidate, with identical inputs and identical context. Pairing is what makes the arithmetic honest. You subtract per example, so the fact that some cases are inherently harder than others cancels instead of swamping the signal.
evalshift all --from gemini-3.1-flash --to gemini-3.1-pro --suite-name checkout-agent
--from
and --to
override defaults.source_model
and defaults.target_model
from your config,
so the file records the migration you are planning while the flags let you audition candidates
without editing it. evalshift all
chains the whole pipeline: doctor β run β evaluate β analyze β report.
Rehearse for free first. evalshift demo
scaffolds a runnable project, and
evalshift all --offline --yes --open
replays canned fixtures through the same pipeline with no API keys and no spend.
## Step 3 β score with more than one lens #
No single scorer catches all three drift classes, and evaluators cost little next to the model calls. Configure several.
structural
evaluators β json_schema
, regex
, length
β are free and make no API calls. Being deterministic makes them the cheapest possible alarm. If your output has any contract, encode it here: a schema that stops validating needs no judgment call.
semantic
compares embeddings. The source output is pinned at 1.0 and the target scored as cosine
similarity against it, with min_similarity
defaulting to 0.9
. That answers "did the meaning move," not "is it better" β useful as a drift alarm, misleading as a quality score.
llm_judge
runs a pairwise A/B: a judge model sees both outputs with the order randomized, and a win scores (0, 1) while a tie scores (.5, .5). The randomization is load-bearing β without it, a judge's preference for whichever answer it read first becomes your migration verdict.
tool_selection
and tool_arguments
cover agents. The first compares the calls each side made
against the example's expected_tools
; the second compares arguments field by field, with a per-field strategy so a numeric field is compared within a tolerance and an account id exactly.
Every evaluator config also takes blocking: bool = true
. Set it to false
and the results are advisory only: they show up in the report and never gate a decision β the right home for a judge criterion you have not yet learned to trust. Full reference: /docs/evaluators.
## Step 4 β read the statistics, not the average #
Deltas are computed pairwise and grouped per (prompt_id
, evaluator_name
, slice_name
), and the first thing the analysis does is refuse questions the data cannot support. Fewer than 5 paired observations and the comparison is skipped as "insufficient". Between 5 and 20 it is tested but flagged uncertain.
The test is chosen rather than assumed: Shapiro-Wilk at Ξ±=0.05 on the deltas picks a paired t-test when they look normal and a Wilcoxon signed-rank test when they do not. Every testable comparison in the run then goes through a Benjamini-Hochberg FDR correction at Ξ±=0.05, because a suite with forty comparisons will hand you two "significant" findings by luck alone if nobody corrects for it.
Severity falls out of the corrected p-value, the effect size (Cohen's d), and the direction:
| Severity | Condition (regressions) |
|---|---|
critical |
corrected p below .01 and effect size above 0.8 |
high |
significant, effect size above 0.5 |
medium |
significant, effect size above 0.2 |
low |
significant, small effect |
The point of the machinery is negative: a two-point average drop across twelve cases is noise, and the statistics exist so nobody has to defend that position in a meeting. The method is written up in /docs/methodology.
## Step 5 β decide with a written policy, not a meeting #
Write the thresholds down before you see results. A migration_policy
block turns the analysis into
one of four verdicts β pass
, conditional_pass
, fail
, or inconclusive
β recorded in
migration_decision.json
. Only blocking evaluators gate it; advisory results are summarized separately and never flip the verdict.
The interesting verdict is inconclusive
. Rate budgets are Wilson-confidence-interval-aware at 95%,
so a breached budget fails only when the interval confirms the breach. A breach the interval still
spans comes back as inconclusive
β "your suite is too small to tell" β rather than a failure you would have overridden anyway. Count, cost and latency budgets are exact and always conclusive.
A fail
means a conclusive budget failure or a blocking critical or high comparison.
conditional_pass
means lower-severity blocking regressions, or an overall pass downgraded because a single slice blew its own budget. See /docs/migration-policy for the config and /docs/verdicts for how each one is computed.
## What this looks like in one afternoon #
- +Instrument your agent with the capture SDK and record a day of real traffic.
- +Run
evalshift capture sync
to turn those captures into a golden suite. - +Run the pipeline offline once, to validate the suite before it costs anything.
- +Run it live against both models, paired.
- +Read the report β start at the severities, not the averages.
- +Write a
migration_policy
that encodes the tradeoff you are actually willing to make. - +Wire the same run into CI so the next model bump is a pull request check instead of an afternoon.
Steps one through six are a one-time cost. The seventh is what keeps them from recurring every quarter.
## Keep reading #
Running LLM regression tests in CIβ the paired run as a pull request check. - + When to trust an LLM judgeβ what pairwise judging is good at, and where it quietly misleads you. - + Getting startedβ install, scaffold, and a first offline run.