{"slug": "build-an-llm-model-deprecation-migration-plan-before-the-cutoff-week", "title": "Build an LLM model deprecation migration plan before the cutoff week", "summary": "OpenAI notified developers on April 14, 2025 that GPT-4.5 Preview would be removed from its API on July 14, 2025, giving teams roughly three months to migrate, according to a Startup Fortune guide on LLM deprecation planning. The guide recommends pinning dated snapshot IDs such as gpt-4o-2024-08-06 or claude-sonnet-4-20250514 in a single config file instead of floating aliases, building a golden set of 150 to 300 real production cases to catch prompt regressions, and keeping the old model reachable for rollback after cutover. It notes Anthropic commits to at least 60 days' notice before retiring a publicly released model and advises planning on a replacement every six to twelve months.", "body_md": "*Your provider will retire the model your product runs on, and the notice you get is shorter than most roadmaps. The teams that cope are the ones that wrote the plan while nothing was on fire.*\n\n- Retirement notices usually give you months, not years: OpenAI announced on April 14, 2025 that GPT-4.5 Preview would be removed on July 14, 2025.\n- Pin dated snapshot IDs in one config file, never a floating alias, so a swap is a one-line change you choose to make.\n- A golden set of 150 to 300 real production cases is the only thing that tells you whether a new model broke your prompts.\n- Shadow traffic and a cost check on real token counts catch the regressions that a test set can't, including bills that rise on the same prompts.\n- Keep the old model reachable for a rollback window after cutover, and put the provider's retirement date on the team calendar the day the notice lands.\n\nOpenAI told developers on April 14, 2025 that GPT-4.5 Preview would be removed from its API on July 14. That's three months from notice to shutdown for a model people had only had access to since February. If your product called it by name, you had a quarter to find every prompt that depended on its behaviour. A written LLM model deprecation migration plan turns that quarter from a scramble into a checklist.\n\nAnthropic's documentation commits to at least 60 days' notice before it retires a publicly released model, and Google and OpenAI publish their own deprecation tables with shutdown dates. Read yours today, because the dates are public and most teams never look until an email arrives. Models don't stay on the shelf for years. Plan on a replacement every six to twelve months, and treat each one as a small release.\n\nWhat follows is the order I'd run it in. None of it needs special tooling, though a few tools make it easier.\n\nStart with an audit. Search your repo for every model string: the obvious one in your API client, but also the one in a background job somebody wrote in March, the one in a notebook that feeds a weekly report, and the one in a customer-facing feature that was shipped as a quick experiment. Startups tend to find three or four more than they expected. Each one is a place where the shutdown date will quietly cause an outage.\n\n[How to handle LLM API rate limits in production before retries burn your quota](https://startupfortune.com/how-to-handle-llm-api-rate-limits-in-production-before-retries-burn-your-quota/)\n\nA 400,000 tokens-per-minute cap, minus 20% held back as headroom, leaves 320,000 to plan against. Splitting that into lanes puts 96,000 on user-facing calls, and a background step costing 18,000 tokens supports only about twelve steps a minute. - [how to handle LLM API rate limits](https://startupfortune.com/how-to-handle-llm-api-rate-limits-in-production-before-retries-burn-your-quota/) - [prevent retries from burning API quota](https://startupfortune.com/how-to-handle-llm-api-rate-limits-in-production-before-retries-burn-your-quota/)\n\nThen move all of them into a single config layer. One file, one mapping from a task name such as 'ticket-summary' or 'invoice-extract' to a model ID and its parameters. Your code asks for the task, not the model. This sounds like housekeeping, and it is, but it's also what makes every later step cheap.\n\nNow the pinning itself. Providers offer two kinds of identifier. A floating alias points at whatever the provider considers current, and a dated snapshot, such as gpt-4o-2024-08-06 or claude-sonnet-4-20250514, points at one frozen version. Use snapshots in production. An alias can change behaviour under you on a date you didn't choose, which is the opposite of what you want from AI model version pinning in production. Aliases are fine in a developer's sandbox, where you want the newest thing by default.\n\nPinning doesn't stop the retirement. It stops surprise. A snapshot lives until its published shutdown date, and you know that date, so the migration happens when you decide, with time to test, instead of when an alias flips.\n\nRecord two more facts next to every pinned ID: the provider's stated retirement date, and the name of the person who owns that task. A deprecation notice that lands in a shared inbox with no owner is how a model dies in production with nobody looking.\n\n## Build the golden set before you need it\n\nA golden set is a fixed collection of real inputs, each paired with what a good output looks like. It's the heart of the plan. Without it, every migration decision comes down to someone reading five outputs and saying they look fine.\n\nBuild it from production, not imagination. Pull a sample of actual requests from your logs, strip or mask personal data, and include the ugly ones: the input with a typo, the one in the wrong language, the one that's 40 times longer than average, the one that made your parser throw last month. Aim for 150 to 300 cases per task. Fewer than that and the numbers lie to you. On 50 cases, a swing from 80% to 90% can be plain noise, and you'll argue about it for a week.\n\nScoring is where teams get stuck. Use the cheapest check that works. For extraction and classification, exact match or field-level comparison against a labelled answer is enough and costs nothing. For anything with a fixed format, validate the schema and count the failures. For free-text outputs such as summaries and replies, a judge model grading against a short rubric works, as long as you spot-check its grades against a human on a few dozen cases. Don't skip that check. A judge that quietly prefers longer answers will wave through a model that's merely wordier.\n\n[How to Calculate AI Agent Cost Per Task Before It Wrecks Your Margin](https://startupfortune.com/how-to-calculate-ai-agent-cost-per-task-before-it-wrecks-your-margin/)\n\nAnthropic's Claude Opus 4.5 pricing runs $5 per million input tokens and $25 per million output, and retries on failed runs can cost more than the original attempt, turning a 70% success rate into roughly triple the cost of a 95% rate. - [how to calculate AI agent cost per task](https://startupfortune.com/how-to-calculate-ai-agent-cost-per-task-before-it-wrecks-your-margin/) - [AI agent pricing model for startup margins](https://startupfortune.com/how-to-calculate-ai-agent-cost-per-task-before-it-wrecks-your-margin/)\n\nOpen-source tools such as promptfoo let you run the same cases against two models and diff the results from a config file, and hosted platforms like Braintrust and LangSmith store the runs so you can compare them across releases. Pick one and wire it into CI. The goal is that anyone on the team can type one command and see old model against new model on the same inputs.\n\nDecide the pass bar before you see any results. Something like: no more than a one-point drop on accuracy, zero new schema failures, and no category of input where the new model does worse by more than five points. If you set the bar afterwards, you'll set it wherever the new model happens to land.\n\n## The prompts will drift, so expect to edit them\n\nHere's what actually breaks. A newer model follows instructions more literally, or less. It adds a friendly sentence before your JSON. It refuses something the old one answered. It reads 'be concise' as three words instead of three sentences. A prompt that was tuned, perhaps unknowingly, around the quirks of one model gets handed to another and loses the thing that made it work.\n\nSo run the golden set on the candidate with your prompts unchanged first. That gives you the raw regression, which is what you'd have shipped had you just changed the ID. Then edit. Change one thing at a time, rerun, and keep the diff. Most fixes are small: restating the output format, moving a rule from the middle of the prompt to the end, adding one example for the case that failed. When a prompt needs a rewrite from scratch, that's useful information too, because it tells you how fragile the old one was.\n\nIf you maintain prompts per model, keep them as separate versions in the same config layer, not as a pile of if-statements in your code. The old prompt stays attached to the old model for as long as you might fall back to it.\n\n## Replay real traffic before it matters\n\nA golden set only knows what you thought to put in it. Production sends things you didn't. Shadow traffic closes that gap: you keep serving users from the old model, and in parallel you send a copy of each request to the candidate and log what it would have said. Nobody sees the candidate's output.\n\nRun it for a week if volume allows, and a few days if it doesn't. Compare the pairs. Look at length distribution first, because it's the quickest signal that behaviour has shifted; if the new model's replies average 30% longer, something changed, and it'll show up in your bill and your UI. Then check failure rates: malformed JSON, empty responses, timeouts, refusals. Then sample the cases where the two outputs differ most and read them yourself. Twenty minutes of reading the biggest disagreements will teach you more than any dashboard.\n\nIf you can't afford to double your inference spend for a week, shadow a slice. Five to ten percent of traffic, chosen at random, is plenty to expose a pattern. After shadowing, move to a canary: send that same slice of live users to the new model, watch support tickets and your own quality metrics, and ramp to 25, 50 and 100 percent only if nothing moves.\n\n## Check the cost delta on real tokens\n\nPrice per million tokens is the number on the pricing page, and it's the least useful one. What you pay is price times tokens, and tokens change between models in ways the page doesn't show. Different model families use different tokenizers, so the same prompt can count as more input tokens on the candidate. Reasoning-style models can spend a large number of output tokens thinking before they answer, and you're billed for them. A model that's cheaper per token can cost more per task.\n\nThe fix is to measure. Your shadow run already logged input and output tokens for both models on identical requests, so multiply by each model's current price and compare cost per completed task. Do the same for latency at the 50th and 95th percentile, since a slower model can break a timeout you set two years ago. If you rely on prompt caching, check that the cache still hits under the new model and that its pricing hasn't changed, because a lot of the savings in long-prompt products come from there.\n\nWhen the new model costs more, you have options besides eating it. Shorten the prompt, cap the output length, route easy requests to a smaller model and keep the larger one for hard ones, or negotiate. But do the arithmetic before the cutover, with your finance person in the room, not after the first invoice.\n\n## How to swap an LLM provider without breaking prompts\n\nSometimes the retirement notice is the push you needed to leave a provider altogether, and it's the harder migration. Prompts that worked on one family often degrade on another, and the differences are practical: system prompt handling, how tool calls and structured outputs are formatted, stop sequences, token limits, and how each model reacts to examples.\n\nPut a thin gateway between your code and the vendors. LiteLLM, which exposes many providers behind one OpenAI-style interface, and OpenRouter, which does the same as a hosted service, are the common choices; some teams write about 200 lines of their own. The point is that switching a task from one provider to another becomes a config change, and your application never imports a vendor SDK directly. That doesn't make the prompts portable, only the plumbing. Prompts still need the golden-set treatment above, run against the new provider, with edits.\n\nIt also helps to keep a second provider warm for your most important task. Even a low-traffic fallback that's tested monthly means a vendor outage, or a shutdown date you misjudged, doesn't take the product down with it.\n\n## Keep a rollback window, and put the date in the calendar\n\nCutting over isn't the finish line. After you move 100% of traffic, leave the old model reachable and your old prompts intact for at least two weeks, longer if its shutdown date allows. A regression that only affects a rare input type won't appear in a day. Keep a flag that flips the task back to the old ID without a deploy, and test that flag once before you need it.\n\nWork backwards from the shutdown date. If the provider retires a model on a Monday, the plan needs to be finished well before it: audit and golden set in the first week, candidate runs and prompt edits in the second and third, shadow and canary across the fourth, full cutover with at least two weeks of rollback margin left. For a 60-day notice, that's already tight, which is the argument for building the golden set and the gateway when nothing is forcing you to. The first migration is the expensive one. The second reuses everything.\n\nA short written runbook is enough: where the model IDs live, who owns each task, where the golden sets are, the pass bar, the flag for rollback, and the retirement dates. Review it each quarter and after every model you adopt. And when the notice email arrives, copy the shutdown date into the team calendar that same hour, with a reminder a month before.\n\nFrankly, nobody enjoys this work, and no customer will ever thank you for it. They'll only notice if you skip it.\n\n**Also read:** [How to Set LLM Eval Pass Threshold Before the Demo Does It for You](https://startupfortune.com/how-to-set-llm-eval-pass-threshold-before-the-demo-does-it-for-you/) • [Europe, EU, European Union and Schengen Explained in 60 Seconds](https://startupfortune.com/europe-eu-european-union-and-schengen-explained-in-60-seconds/) • [Write an AI agent escalation policy before customers hit a dead end](https://startupfortune.com/write-an-ai-agent-escalation-policy-before-customers-hit-a-dead-end/)\n\n*This article is posted in [AI News](https://startupfortune.com/category/ai/), check it out for more related stories.*\n\n## Join the discussion\n\n[Open in the community →](https://startupfortune.com/community/)\n\nAlmost there. Sign in and your reply posts straight away.", "url": "https://wpnews.pro/news/build-an-llm-model-deprecation-migration-plan-before-the-cutoff-week", "canonical_source": "https://startupfortune.com/build-an-llm-model-deprecation-migration-plan-before-the-cutoff-week/", "published_at": "2026-11-21 00:56:10+00:00", "updated_at": "2026-11-21 01:12:43.957591+00:00", "lang": "en", "topics": ["large-language-models", "ai-products", "ai-tools", "mlops", "developer-tools"], "entities": ["OpenAI", "GPT-4.5 Preview", "Anthropic", "Google", "Startup Fortune", "gpt-4o-2024-08-06", "claude-sonnet-4-20250514"], "also_reported_by": [], "takeaway": "Pin dated snapshot IDs like gpt-4o-2024-08-06 in one config file rather than floating aliases, and validate each swap against a golden set of 150 to 300 real production cases before cutover.", "recommendation": {"advice": "Pin dated snapshot IDs in a single config file and keep the retired model reachable for rollback; test migrations against 150 to 300 real production cases plus shadow traffic before cutover.", "playbook": "production-ops", "playbook_url": "https://wpnews.pro/playbooks/production-ops"}, "alternates": {"html": "https://wpnews.pro/news/build-an-llm-model-deprecation-migration-plan-before-the-cutoff-week", "markdown": "https://wpnews.pro/news/build-an-llm-model-deprecation-migration-plan-before-the-cutoff-week.md", "text": "https://wpnews.pro/news/build-an-llm-model-deprecation-migration-plan-before-the-cutoff-week.txt", "jsonld": "https://wpnews.pro/news/build-an-llm-model-deprecation-migration-plan-before-the-cutoff-week.jsonld"}}