# A New Model Dropped. Don't Just Swap the ID. > Source: > Published: 2026-09-21 02:48:33+00:00 The new model is out. The benchmarks look great. Your provider has already scheduled the sunset date for the one you are running. So you swap the model ID, run a few requests, see nothing obviously broken, and ship. I want to argue that this is one of the most common ways LLM-backed systems quietly degrade — and that it tends to happen not because teams are careless, but because "the model works" and "the model is safe to run in production" are two different claims, and only the first one is cheap to check. Ask any engineer whether you should evaluate a new model before switching to it. Most will say yes. Then look at what often happens when the deprecation email arrives. A recent paper on model migration in production systems put the problem plainly: deprecation and migration cycles come around roughly every twelve months, they hit every product built on the retired model, and task-specific evaluation is essential — but a manual evaluation process is too slow and too expensive to fit inside the migration window ([Casey et al., 2026](https://arxiv.org/abs/2604.27082)). The result is easy to guess. Evaluation tends to get skipped, then done anyway in the last week before the sunset, in a panic, on whatever inputs someone had lying around. The gap is not one of belief. It is one of process cost. Few people skip a test suite that runs in a minute. Many skip the one that takes three days. Part of why the swap feels safe is the release announcement. The new model scores higher on the public benchmarks, so it should score higher on your task too. That inference is where things can go wrong. A new model version might not be a backward-compatible upgrade of the old one. It may be a different model, trained differently, with different habits, and nobody has checked that against your prompt. The prompt you tuned against the old model's habits carries no guarantee with the new one — the same prompt is not promised to perform as well, let alone better. And the reported gains were measured on public datasets that look nothing like your contracts, your invoices, or your support tickets. "Better on the benchmark" and "better on your data" are correlated, not equivalent. The only way to know the second one is to measure it. When you do measure it, three things can move, and they can move in opposite directions. **Accuracy.** The new model may be better on average and worse on the field you care about. It may handle the easy cases identically and change its mind on the ambiguous ones — which are exactly the cases your spot check did not include. **Latency.** Newer models are frequently slower per request, particularly the ones that spend tokens reasoning before they answer. A jump in median latency from 1.5 seconds to 3 seconds is easy to miss in a manual test and hard to miss in a UI that blocks on the response. **Token volume.** This is the one that tends to surprise people. A new model with the same prompt will often produce longer output: more hedging, more explanation, more fields filled in that you asked it to leave empty. Output tokens are the expensive ones, and they compound. A model that is 20% cheaper per token and produces 80% more of them is a price increase. Here is the kind of comparison that decides an upgrade — same prompt, same frozen dataset, old model versus new: ``` model-a (current) model-b (candidate) accuracy (overall) 91.2% [89.0 – 93.4] 92.8% [90.7 – 94.9] accuracy (weak field) 74.5% [70.1 – 78.9] 81.0% [77.0 – 85.0] latency p50 / p75 1.4s / 1.9s 2.6s / 3.8s output tokens p50 180 410 ``` Read that table honestly. The headline accuracy gain is inside the noise — the intervals overlap, so you have not shown an improvement. The weak field did get better, and that might be worth a lot. But latency nearly doubled and output volume more than doubled. Whether to ship this is a real decision with real trade-offs, and you cannot make it from "I ran it and it looked fine." The point of measuring three dimensions is to have a rule, so the upgrade is not a matter of who argued loudest in the meeting. | Dimension | Compare | Ship if | |---|---|---| | Accuracy | Confidence intervals, per field | No field's interval moved down below the old one | | Latency | p50 and p75, not the mean | Both stay under your budget; p75 is what users feel | | Token volume | Median output tokens | Within the budget you set when you priced the feature | Three notes on applying it. **Compare intervals, not point estimates.** LLM output is stochastic. A single run of your test set is one sample. Twenty runs give you an interval, and the interval is what tells you whether 91.2% versus 92.8% is signal or noise. **Watch the tails, not the average.** Mean latency hides the slow requests. p75 is the request a quarter of your users wait for. **If any dimension regresses, the answer is "not yet", not "no".** In my experience, most regressions on a model upgrade turn out to be prompt-model mismatches rather than model failures. The prompt was tuned against the old model's habits. A shorter, stricter output instruction often brings token volume back down; a re-tuned prompt often recovers the field that dropped. But you only know what to fix if you measured it. None of the above is intellectually hard. The reason it tends not to happen is that doing it by hand often looks like this: Each step is a few hours. Together they can add up to a week nobody has during a migration window. So the process gets compressed into "swap the ID and watch the error rate", which catches crashes and can miss everything in the table above. This is the workflow [PromptProof](https://promptproof.hatchandco.cloud/) exists for. You keep a labeled dataset in the product, shared with the team, so the domain expert who knows the right answers can maintain it and it does not go stale on someone's laptop. When a new model arrives, you attach the same prompt to the old model and the new one and run a single [statistical experiment](https://promptproof.hatchandco.cloud/features/experiments/): 10–50 trials per item, a confidence level you choose, executed in the background. What comes back is the table above, without the script: It works across OpenAI, Anthropic, and Google models in the same experiment, so "should we move to the new model" and "should we move to a different provider" are the same comparison. The intent is not that the tool decides for you. It is that the three numbers you need for the decision cost you minutes of setup instead of days, so the evaluation is much more likely to happen before the sunset date instead of after the support tickets. The next model will probably be better than this one. That is the whole reason to measure before you switch — because the way it is better is often not the way you assumed. *How does your team decide when to move to a new model? I would like to hear what you measure, and what you have been burned by not measuring.*