This one passed: 71% cheaper, and the agent still does the same thing

EvalShift's migration report recorded a PASS verdict for a customer-support agent moved from gemini-3.7-pro to gemini-3.7-flash, with the candidate 71.2% cheaper and 61.9% faster while staying inside every budget in the migration_policy block of evalshift.yaml. The policy set a 5% max overall regression rate, zero critical regressions, 90% minimum equivalence, 10% tool-argument drift, 5% tool divergence, 0% cost increase and 30% latency increase, with the security slice tightened to 0% regression and 0% tool divergence and the refund slice to 0% tool-argument drift. The author notes a rate over n rows moves only in steps of 1/n, so the suite's 108 tool-argument rows make a 1% budget a zero-tolerance budget in disguise.

← all posts https://www.evalshift.dev/blog This one passed: 71% cheaper, and the agent still does the same thing An EvalShift migration report that said PASS: a 71% cheaper model inside every policy budget. Where each threshold came from, and what PASS does not claim. The previous post was about a migration that was cheaper, faster and still failed. This is the other outcome. It is the more common one once a suite is in decent shape, and I see it written up far less often, because a pass is boring. It shouldn't be. A pass is only worth something if the limits were fixed before the run. So this post is the whole thing: the policy, the reasoning behind every number in it, and the EvalShift report it produced. For anyone arriving here cold: EvalShift is a local-first CLI for testing LLM model migrations. It replays a frozen golden suite against your current model and a candidate, scores every pair of outputs with tool-call, structural, semantic and LLM-judge evaluators, runs paired statistics over the deltas, and turns a migration policy block you wrote in evalshift.yaml into one of four verdicts: pass , conditional pass , fail or inconclusive . It writes a single-file report.html on your machine; nothing is uploaded unless you run evalshift push . Every figure in this post is a panel from that report. The candidate: a customer-support agent with six tools lookup customer , lookup order , check refund policy , issue refund , escalate to human , search kb , moving from gemini-3.7-pro to gemini-3.7-flash . The suite: 120 examples recorded in production by the EvalShift capture SDK, promoted into a golden JSONL suite with evalshift capture sync , and sliced by tag according to what the conversation was about. | Slice | Examples | What is in it | |---|---|---| | routine | 42 | order status, shipping, account questions | | refund | 26 | refund and return requests | | security | 24 | account access, password and payment-method changes | | customer lookup | 16 | requests that need a customer record first | | text only | 12 | greetings, thanks, off-topic | One evalshift compare command, real API calls on both sides, and the report opened on: Cheaper by 71.2%, faster by 61.9%, verdict PASS. The economics look like the last post's. The difference is everything that was decided before the run. The numbers were written before the run EvalShift's migration policy is a block in evalshift.yaml with seven budgets: overall regression rate, critical regression count, equivalent-or-better rate, tool-argument drift, tool-selection divergence, cost increase and latency increase. A slices map under it lets any slice override any budget, inheriting the top-level value where it doesn't. Evaluators are configured per suite; budgets are set once and tightened per slice. evalshift init --profile cost-reduction scaffolds a starting policy: 2% overall regression rate, zero critical regressions, 97% equivalence, 1% tool-argument drift, 5% cost increase, 30% latency increase. That is a starting point, not a decision. This is what the project actually ran with: migration policy: max overall regression rate: 0.05 max critical regressions: 0 min equivalence rate: 0.90 max tool argument drift: 0.10 max tool divergence: 0.05 max cost increase: 0.0 max latency increase: 0.30 slices: security: max overall regression rate: 0.0 max tool divergence: 0.0 refund: max tool argument drift: 0.0 Three things moved between the profile and this file. Check every rate against the suite size A rate over n rows can only move in steps of 1/ n . This suite has 108 tool-argument rows, so one drifted row is 0.93%, and a 1% budget is a zero-tolerance budget wearing a percentage. EvalShift prints a recommendation when a budget is below the granularity of its denominator, naming the budget, the value and the row count, and this one would have triggered it. So the rule I use: if I mean zero, I write 0.0 . Where I don't, I set a number the sample can resolve. 10% argument drift on 108 rows is ten reworded search queries, which is what drift on this agent looks like in practice. Tolerance where wording lives, zero where money and access live The overall regression rate went up , from 2% to 5%. The pairwise LLM judge is blocking on this project, and a judge flips on rewording; 2% of 444 comparisons is nine judge calls deciding the migration. 5% leaves room for the judge to disagree about phrasing without anything real being allowed through. The strictness moved into the slices instead. security gets zero regressions and zero tool-selection divergence: a model that starts routing an account-access request to a different tool does not get a percentage. refund gets zero argument drift: an order id or an amount that drifts is a wrong refund, not a reworded one. The slices that are not in the policy matter too. customer lookup has 16 examples. EvalShift tests a comparison below 20 pairs but flags it uncertain, so that slice inherits the top-level budgets and gets no tighter ones. It is the slice I would grow before I tightened it. The cost budget is the reason for the migration max cost increase: 0.0 . The point of the exercise is to spend less. A candidate that costs more has failed before any quality number is read, and a policy should say so instead of leaving it to whoever reads the economics card. EvalShift measures cost and latency from the run's own calls, so these two budgets gate even when no quality evaluator does. Two evaluator decisions sit alongside the budgets. Every EvalShift evaluator carries a blocking flag; advisory blocking: false results are reported and ranked but never change the verdict. The judge is blocking: true here; init writes false because at a dozen examples judge noise would decide the verdict, and at 120 with an audited criterion it earns its vote. semantic stays advisory. It measures wording, and wording is the one thing this migration was allowed to change. What the run measured | Budget | Scope | Observed | Limit | |---|---|---|---| | Overall regression rate max overall regression rate | overall | 3.6%16 of 444 · 95% CI 2.2–5.8% | ≤ 5.0% | | Critical regressions max critical regressions | overall | 0of 444 | ≤ 0 | | Equivalent-or-better rate min equivalence rate | overall | 96.4%428 of 444 | ≥ 90.0% | | Tool-argument drift max tool argument drift | overall | 4.6%5 of 108 tool-argument rows | ≤ 10.0% | | Tool-selection divergence max tool divergence | overall | 2.8%3 of 108 divergence rows | ≤ 5.0% | | Cost increase max cost increase | overall | 0.0%cost fell 71.2% | ≤ 0.0% | | Latency increase max latency increase | overall | 0.0%latency fell 61.9% | ≤ 30.0% | | Overall regression rate max overall regression rate | security | 0.0%0 of 96 | ≤ 0.0% | | Tool-selection divergence max tool divergence | security | 0.0%0 of 24 | ≤ 0.0% | | Tool-argument drift max tool argument drift | refund | 0.0%0 of 26 | ≤ 0.0% | The first row is the one worth reading twice. 16 of 444 is 3.6%, and its 95% Wilson interval runs from 2.2% to 5.8%. The interval crosses the limit. EvalShift's rule is asymmetric on purpose. A breach fails only when the interval confirms it; a breach the interval cannot confirm returns inconclusive , because the suite was too small to say. A budget the observation held is conclusive however wide its interval, because a wide interval must never downgrade a clean run. This budget held, so it passes, and the interval is printed so the reader knows how much room there was. The three slice budgets held at exactly zero. Zero on 96 comparisons means something. Zero on 12 would not, which is why text only has no slice budget at all. By evaluator Four evaluators scored this run. EvalShift's tool selection evaluator reads the recorded traces and scores two axes: conformance, where each side is graded against the suite's recorded tool calls, and divergence, where the target is graded against what the source did. tool arguments scores argument values field by field against the expected call. The llm judge is pairwise and sees the two outputs as anonymous A and B. semantic is embedding similarity between the two outputs. The first three are blocking; the last is advisory. | Evaluator | n | Score Δ | Effect | 95% range | Conf. | Result | |---|---|---|---|---|---|---| | Routing — conformance routing · tool selection.conformance | 108 | +0.046 | 0.28small | 0.09, 0.47 | Likely | ✓ ImprovedTarget scores higher than source. | | Routing — divergence routing · tool selection.divergence | 108 | -0.028 | 0.17negligible | -0.36, 0.02 | Unclear | ✓ EquivalentNo meaningful difference between models. | | Routing args routing args | 108 | -0.004 | 0.03negligible | -0.22, 0.16 | Unclear | ✓ EquivalentNo meaningful difference between models. | | LLM judge: equivalence llm judge.equivalence | 120 | +0.029 | 0.12negligible | -0.06, 0.30 | Unclear | ✓ EquivalentNo meaningful difference between models. | | Semantic similarityadvisory semantic | 120 | -0.041 | 0.44small | -0.62, -0.26 | Likely | ✗ Regressed — mediumReported, not gating: blocking is false. | Four blocking rows say equivalent or improved. The one that regressed is advisory, and it measures the one thing this migration was allowed to change. Had semantic been blocking, the same run would have come back conditional pass on a medium-severity regression in phrasing. That is not a reason to make every evaluator advisory. It is a reason to decide, per evaluator, whether what it measures is something you are willing to block a migration on. The conformance row says the candidate matched the suite's recorded tool calls more often than the model that produced them: 11 examples improved, one regressed. Most of the eleven were refund requests where the source went straight to issue refund . The diffs I still read A pass is not permission to skip the diff. For every flagged example, the EvalShift report shows the reason it was flagged, the tool calls on each side, the argument-level diff, and the conversation context that led to it. Two examples from this run, one from each side of the ledger. The refund went out either way, and the final text on both sides would pass any output check you care to write. The trace is where the difference lives: one side verified before acting and the other did not. This is the class of change a text evaluator cannot see in either direction, and the reason EvalShift's tool-call evaluators score the trace rather than the prose. A reviewer would call those the same query. The evaluator scored it 0.84 against a 0.9 floor and counted it as drift, five rows out of 108 did the same, and the budget said five is fine. That is what the 10% was for. The same five rows in the refund slice would have failed the run, and that was also decided in advance. What EvalShift did in this run The whole run, as a list of the product's parts, in the order they were used: - + The capture SDK recorded the agent's real conversations in production, tool calls included, and evalshift capture sync promoted them into a frozen golden JSONL suite with the tool evaluators written from what the captures actually contained. - + evalshift compare replayed every example against the source and the target model, paired per example, with the same inputs, tools and context on both sides. - + The tool-call evaluators tool selection , tool arguments scored the traces, the pairwise LLM judge scored the outputs, and semantic measured drift in wording, advisory only. - + Paired statistics turned each evaluator's deltas into an effect size, a 95% interval and a corrected confidence label, so a two-point average drop and a real regression are told apart mechanically. - + migration policy in evalshift.yaml held seven budgets, three of them tightened to zero on the security and refund slices, and every proportion budget was judged with a Wilson interval that can return inconclusive instead of a false fail. - + report.html was written locally, verdict first, with a per-example diff for everything flagged. Nothing left the machine. - + --policy-gate made the verdict an exit code, which is what the EvalShift GitHub Action uses to block a pull request on the same policy once the suite runs in CI. What a pass proves Not that nothing changed. 16 comparisons regressed and 28 improved; the agent phrases search queries differently and checks the refund policy more often than it used to. It proves that every change stayed inside limits that were written down before anyone saw a number, on a suite that was frozen before the run. That is the entire claim, and it is enough to act on, because there is nothing left to negotiate: the argument about what counts as acceptable happened in the YAML, not in the meeting after the report. What happens next is the boring part, which is the point. The model string changes in production. The suite does not. It runs again on the next pull request through the EvalShift GitHub Action, gated on the same policy, against the new baseline. evalshift compare --suite-name support routing --to gemini-3.7-flash --policy-gate --open If the previous post was the reason to run the comparison, this one is the reason to write the policy first.