How to Roll Out a New LLM Model Version Safely

A new large language model (LLM) version can alter production behavior in unpredictable ways, so organizations should use a gated, progressive, reversible rollout that includes defining a production contract, testing against realistic and adversarial workloads, comparing the candidate with the incumbent, shadow-mode observation, gradual live exposure, and a tested rollback or human fallback. The guidance, presented by Burns at USENIX SREcon25 Americas, emphasizes measurable functional and safety requirements, such as at least 99.5% schema validity in critical workflows, and references the OWASP Top 10 for LLM Applications to structure security risks.

Why Does DNS Still Point to the Old Server After the TTL Expires? DNS, or the Domain Name System, maps a hostname such as www.example.com to an IP address. A DNS time to live TTL https://www.rfc-editor.org/info/rfc1035 … A new large language model LLM version is not a routine dependency update. It changes the behavior https://www.usenix.org/conference/srecon25americas/presentation/burns of the production system that uses it. The new version might improve accuracy while producing invalid JSON, selecting different tools, refusing valid requests, increasing latency, leaking information through a retrieval path, or failing cases that the previous version handled correctly. Use a gated, progressive, reversible rollout . Define the behavior that must remain true, test the candidate against realistic and adversarial workloads, compare it with the current model, observe it in shadow mode, expose it to live users gradually, and keep a tested rollback or human fallback available. A candidate model is the version being evaluated. The incumbent is the version currently serving production traffic. Comparing those two versions at the level of individual tasks and high-risk cases is more useful than relying on a general benchmark score. Define the Production Contract Write down the production contract before testing the candidate. This contract states what the LLM-powered service must do, what it must never do, and which performance limits it must meet. A benchmark can show that one model performs better on a general task. It cannot establish that the model meets the requirements of your workflow. Production requirements depend on the surrounding prompt, retrieval pipeline, tools, permissions, parsers, safety controls, and user population. Set Functional Requirements Specify the supported tasks and the expected behavior for each one. Include requirements for: - Output fields, formats, and schemas - Retrieval and citation behavior - Tool selection and tool arguments - Handling of ambiguous or incomplete requests - Refusal and escalation behavior - Multi-turn context - Maximum latency and timeout rates - Cost per request or completed task - Availability and rate-limit handling Turn each important requirement into a measurable test. For example, a structured customer-support workflow might require schema-valid output, a valid ticket category, a grounded answer when company documentation is available, and escalation when the request requires human approval. Use separate release gates for separate risks. An illustrative contract might require at least 99.5% schema validity in a critical workflow, no unauthorized destructive tool calls in adversarial tests, and no increase in critical-severity policy violations. These numbers are examples, not universal standards. The appropriate thresholds depend on the harm, reversibility, and business impact of failure. Map Safety and Security Risks Document prohibited outputs and actions alongside ordinary functionality. Cover risks such as: - Direct jailbreak attempts to bypass safety instructions - Indirect prompt injection in retrieved documents, webpages, emails, or files - System-prompt or credential extraction - Cross-user or cross-tenant data leakage - Unauthorized tool calls - Malicious file handling - Excessive-cost or context-exhaustion requests - Unsafe behavior after a tool or dependency failure The OWASP Top 10 for LLM Applications https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025 provides a useful structure for organizing these threats. For each external action, define the permission boundary. The model should not be the sole authority for sending a message, changing a record, approving a payment, or deleting data. Require deterministic authorization checks and human approval for sensitive actions where appropriate. Also document privacy and retention rules. Record which user data enters the model context, which provider receives it, how long traces remain available, and who can inspect evaluation samples. De-identify and privacy-review production examples before using them for testing. Document the Incumbent Create a reproducible record of the deployed configuration: - Model name, provider, endpoint, and snapshot - System and developer instructions - Inference parameters - Safety settings - Retrieval model, index, corpus, and reranker - Tool definitions, permissions, and schemas - Output parsers and post-processors - Latency, error, cost, refusal, and escalation metrics - Existing known failures and incidents The incumbent baseline lets you identify negative flips , which are cases that the old model handled correctly but the new model handles incorrectly. Research on successive LLM versions found that these regressions can appear even when overall performance improves. The MUSCLE compatibility study https://arxiv.org/abs/2407.09435 treats preserving prior behavior as an explicit objective for updates. Build Production-Based Evaluations Evaluate the complete LLM workflow, not just the model’s raw answers. A candidate that performs well in an isolated test can still fail after retrieval, tool calling, parsing, routing, and policy enforcement are included. Compare every candidate result with the incumbent result where possible. Record whether each version completed the task, violated a constraint, required escalation, produced a side effect, or failed operational limits. Combine Multiple Test Layers Separate testing into layers so that one strong score does not hide a local regression. 1. Application contract tests Validate schemas, required fields, permissions, routing, timeouts, retries, and parser behavior. 2. Capability tests Measure extraction, classification, summarization, reasoning, drafting, or other supported tasks using domain-specific ground truth. 3. Safety and security tests Test refusals, privacy boundaries, jailbreak resistance, prompt-injection handling, and protection against unauthorized actions. 4. Workflow tests Exercise retrieval, citations, multi-turn state, tool selection, tool errors, escalation, and recovery. 5. Operational tests Measure latency percentiles, throughput, token use, cost, rate limits, context failures, and serving capacity. Run deterministic checks wherever the expected result is objective. Validate JSON with a parser, check citation identifiers against retrieved sources, compare tool permissions against an allowlist, and verify that the candidate cannot bypass authorization through natural-language instructions. Layer-specific tests catch failures that aggregate metrics miss. A recent layer-isolated evaluation study https://arxiv.org/abs/2606.11686 reported large drops in individual regression slices while aggregate pass rates changed only modestly. That result comes from a preliminary study of one production agent, so it is not a universal effect size, but it supports separating deterministic application tests from open-ended model-quality tests. Replay Representative Traffic Build the evaluation corpus from several sources: - Curated golden cases with expected behavior - Privacy-reviewed and de-identified production examples - Past incidents and security findings - Support complaints and corrections from users - Long conversations and incomplete requests - Adversarial prompts - Tool-use trajectories - Different languages, regions, user groups, and workflows Keep a held-out set that the release process does not repeatedly optimize against. Refresh the set when production reveals a verified failure. Replay realistic inputs in a controlled environment without triggering side effects. Include the same retrieval data, tool schemas, preprocessing, and output validation used in production. If the candidate receives different context from the incumbent, record that difference rather than attributing every output change to the model. Production behavior can shift substantially between model versions. One study found that GPT-4 accuracy on a prime-versus-composite task fell from 84% to 51% between versions released in March and June 2023, alongside changes in code formatting, instruction following, and handling of sensitive questions. The study of changing ChatGPT behavior https://arxiv.org/abs/2307.09009 demonstrates why a model identifier alone is not a sufficient compatibility guarantee. Use Hybrid Evaluation Use the right evaluator for each property: - Deterministic checks for schemas, citations, exact values, permissions, tool names, and prohibited actions - Standard metrics where reliable ground truth exists - LLM judges for open-ended qualities such as relevance or completeness - Human review for ambiguous, novel, high-impact, or safety-sensitive cases An LLM judge is a model that scores another model’s output against a rubric. It can reduce review cost, but it is not automatically objective. The NeurIPS review of LLM judges https://proceedings.neurips.cc/paper files/paper/2025/hash/829e8f32d76a6248815bf5b01633811d-Abstract-Position Paper Track.html identifies unresolved questions about their validity, reliability, bias, and cost. Calibrate the judge against expert-labeled examples. Version the judge model, rubric, and scoring prompt independently from the candidate. Re-run calibration whenever the judge changes. Do not allow an unvalidated judge to approve high-severity safety cases by itself. Report both absolute and relative results: - Did the candidate meet the minimum requirement? - Did it improve or regress against the incumbent? - What is the negative-flip rate? - Which critical cases failed? - Which user or workflow segments changed? - How much uncertainty surrounds the estimate? A candidate must pass absolute gates, relative regression gates, and mandatory critical-case gates. Average quality improvement cannot offset a prohibited privacy leak or unauthorized action. Test Critical Failure Modes Rare failures require targeted tests because average quality metrics underrepresent their severity. Test the complete application with malicious inputs, untrusted retrieved content, faulty tool responses, and long multi-turn interactions. Run Security and Safety Tests Include direct and indirect attacks: - Ask the model to ignore its operating instructions. - Place injection instructions inside retrieved documents. - Request system instructions, credentials, or private context. - Attempt to access another user’s data. - Supply malicious or malformed files. - Ask for unauthorized tool calls. - Create long prompts designed to exhaust context or cost limits. - Use multiple turns to gradually change the model’s operating context. - Return tool errors, contradictory results, or missing fields. Test both the model response and the enforcement layer. A refusal is not enough if the request still reaches a privileged tool. Conversely, a safe model response does not prove that a parser, router, or retrieval component preserves the same boundary. Use sandboxed accounts and synthetic data for adversarial testing. For destructive or externally visible actions, assert that the action was blocked rather than merely checking the text of the model’s answer. The NIST Generative AI Profile https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf recommends incident planning, continuous monitoring of third-party generative AI services, and testing rollover and fallback technologies. Review High-Risk Regressions Review every failure in a mandatory risk category even when aggregate scores improve. Classify failures by severity and exploitability: - Critical: privacy disclosure, cross-tenant access, unauthorized destructive action, or severe safety violation - High: incorrect action in a sensitive workflow, persistent prompt-injection success, or major loss of grounding - Moderate: material formatting, refusal, retrieval, or task-completion regression - Low: stylistic changes with no effect on the task or safety boundary Set a release rule before looking at results. For example: any critical failure blocks promotion; high-severity failures require documented remediation or approval; moderate regressions require an accepted impact assessment; and low-severity changes receive monitoring. Do not optimize for the candidate’s best score alone. Preserve the incumbent output and the evaluator’s reasoning for each critical case so reviewers can inspect the exact behavioral change. Shadow the Candidate Model A shadow deployment sends a copy of production requests to the candidate while the incumbent continues serving the user. The candidate output is logged and evaluated, but it is not displayed and cannot create real side effects. Shadowing https://arxiv.org/abs/2607.07184 shows behavior that offline data misses, including input-length distributions, unusual retrieval results, concurrency, provider throttling, and real tool-selection patterns. Compare Realistic Workloads Compare the candidate and incumbent across: - Workflow and task type - User segment and customer tier - Language and region - Input and context length - Retrieval corpus - Tool or integration - Time of day and load level Inspect output changes that ordinary success metrics may miss: - Different rates of refusal or escalation - Invalid formatting - More verbose or repetitive answers - Unsupported citations - Different tool selection - Missing follow-up questions - Changes in latency or token use Use paired comparisons when the same request can be evaluated by both versions. Preserve enough trace context to explain whether a difference came from the model, retrieval results, tool responses, or post-processing. Keep Shadow Actions Safe Give the candidate simulated, read-only, or sandboxed tools. Never allow shadow traffic to send messages, modify records, make purchases, delete data, or trigger irreversible workflows. If the candidate produces a tool call, record and validate it without executing it. For tools that return stateful results, use deterministic fixtures or safely resampled responses so the candidate and incumbent face comparable conditions. Shadowing does not show how users adapt to a different answer. A user may ask a different follow-up question, trust a response more, or act on a new recommendation. Treat shadow results as a promotion signal, then validate user-facing behavior through a controlled canary. Canary the Live Release A canary routes a small, controlled portion of live traffic to the candidate while the incumbent remains available. Expand only when technical, behavioral, safety, cost, and user-impact metrics remain within the release contract. Google Cloud’s reliability guidance https://docs.cloud.google.com/architecture/framework/perspectives/ai-ml/reliability describes canary deployment as a way to establish reliability with a small subset of production traffic before expanding. The percentages used in a rollout are implementation choices, not universal standards. Choose Risk-Based Cohorts Start with internal users or low-risk workflows when possible. Then expose a small external cohort before increasing traffic. Exclude workflows with irreversible side effects until the candidate has passed stronger controls. Segment the canary by factors that could hide localized harm: - Workflow - Language - Geography - Customer tier - Retrieval corpus - Tool or integration involved - Sensitive use case - Client or device A candidate that passes in aggregate can still fail for a language group, a specific integration, or a high-risk workflow. Make those segments visible in dashboards rather than relying on a single global average. Require Gates Between Stages Use explicit approval gates between stages such as 5%, 25%, 50%, and full traffic. Adjust the stages and observation periods to request volume, potential harm, and the time needed to detect rare failures. A low-volume, high-risk workflow needs more evidence per stage than a high-volume, reversible one. At each gate, record: - Traffic volume for the candidate and incumbent - Technical metrics - Quality and task-completion results - Safety and security events - Cost and token changes - Segment-level regressions - Open incidents and unresolved uncertainty - Approval by the designated release owner Pause expansion when a critical event occurs, a mandatory threshold fails, or the evidence is incomplete. Google Cloud’s canary deployment documentation https://docs.cloud.google.com/deploy/docs/deployment-strategies/canary describes staged traffic increases followed by a stable full-traffic phase, but the precise schedule should match the risk of the service. Monitor Quality and Harm Availability and latency do not show whether users receive better or safer results. Monitor technical health alongside behavioral quality, security events, cost, distribution changes, and real user outcomes. Track Technical Health Track at least: - Request errors and timeouts - Retry rates - p50, p95, and p99 latency - Token usage - Cost per request and per completed task - Context-window failures - Rate-limit events - Serving capacity - Retrieval, tool, and dependency failures Compare the candidate with a concurrent incumbent control where possible. A provider-wide slowdown, a new user mix, or a retrieval outage can affect both models, while a candidate-specific regression should affect the candidate disproportionately. Track Behavioral Outcomes Measure whether the service completes useful work: - Task completion - Groundedness and citation correctness - Structured-output validity - Refusal and abstention rates - Escalation rates - User corrections and re-prompts - Complaints and abandonment - Satisfaction - Human-review outcomes Use human review for a sample of high-impact outputs, especially when no reliable ground truth exists. Track the reason for each escalation or correction. A rising re-prompt rate often signals a quality or instruction-following regression even when the request technically succeeds. Also monitor safety and security outcomes, including policy violations, prompt-injection detections, sensitive-data disclosure, unauthorized tool calls, and cross-tenant access violations. A low-frequency event still requires immediate response when the severity is high. Detect Drift and Incidents Watch for changes in: - User mix and request topics - Prompt and context length - Languages and domains - Retrieval corpora and indexes - Tool schemas and dependency behavior - Provider endpoints or safety layers - Evaluation judges and rubrics Preserve representative traces under the approved retention policy. Remove or protect sensitive data before sharing traces for debugging. When an incident is verified, add a minimized reproduction to the regression suite and classify it by affected layer. NIST’s AI Risk Management Framework core guidance https://airc.nist.gov/airmf-resources/airmf/5-sec-core calls for post-deployment monitoring, user feedback, appeal and override mechanisms, incident response, recovery, and change management. Monitoring should therefore continue after full promotion, not stop when the canary reaches 100%. Prepare Rollback Before Launch Rollback is a tested operational capability, not a document written during an incident. Define the triggers, decision owner, traffic-routing procedure, and fallback path before the canary begins. Define Automatic Triggers Pause or roll back automatically for conditions such as: - A critical safety or privacy event - An unauthorized external action - A material increase in high-severity policy violations - Severe schema or parser failures - A statistically credible regression in a critical workflow - A large increase in errors or timeouts - A hard cost or token-use limit breach - A provider outage or an unexpected change in behavior - Segment-specific harm above the approved tolerance Use absolute and relative gates together. The candidate must meet the minimum production contract and avoid material deterioration against the incumbent. A candidate that improves average completion but causes a privacy breach fails the release. Test the Recovery Path Before launch, rehearse rollback in an environment with production-matched routing. Verify that operators can: 1. Route traffic back to the incumbent. 2. Restore the incumbent prompt and inference settings. 3. Restore compatible tools, retrieval configuration, and parsers. 4. Disable external actions. 5. Preserve logs and traces for investigation. 6. Route critical work to a simpler model or manual process. 7. Confirm that queued jobs do not execute with the candidate after rollback. Keep the incumbent available through the defined rollback window. NIST guidance https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf specifically supports testing rollover and fallback technologies, including manual processing when automated behavior is unsafe. Govern Ongoing Model Changes The deployed behavior depends on more than the model name. A provider can change serving infrastructure, safety layers, routing, or an endpoint’s behavior https://arxiv.org/abs/2604.27789 without requiring you to change your surrounding code. Version the Full System Record every component that affects an output: - Model name and immutable snapshot, if available - Provider and endpoint - System and developer prompts - Inference parameters and tokenizer - Safety settings and policy filters - Tools, descriptions, permissions, and schemas - Retrieval and embedding models - Vector index, reranker, and knowledge corpus - Preprocessors, parsers, and post-processors - Evaluation judge, rubric, and test-set version - Routing rules and user cohorts - Serving framework and infrastructure image Store the configuration with the release record. Pin versions where the provider supports immutable snapshots. When the provider offers only a continuously updated service, run scheduled compatibility tests and maintain a fallback provider or non-LLM path for critical workflows. Research on evolving LLM APIs https://arxiv.org/abs/2311.11123 describes how silent provider updates can cause performance regressions and complicate conventional regression testing. Treat provider notifications and observed behavior changes as supply-chain events that require evaluation. Close the Rollout Loop Continue monitoring after full promotion and keep the old version available until the rollback window ends. Review provider notices and compare scheduled evaluation results. Investigate unexplained changes in latency, refusals, formatting, retrieval use, or tool selection. Conduct a post-release review that answers: - Which release gates passed or failed? - Which segments changed? - Which incidents or near misses occurred? - Did rollback work as rehearsed? - Which tests failed to predict production behavior? - Which new failures belong in the permanent regression suite? Full traffic alone does not justify retiring the incumbent. Retire it after the service has demonstrated stable behavior, the rollback window has closed, and a tested fallback remains for critical workflows. Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request. Take a look at vroni.com