How to safely roll out changes to a customer-facing AI agent: an enterprise checklist Agnost AI, a company that builds production monitoring for customer-facing AI agents, published a five-stage checklist for safely rolling out agent changes: replaying changes against real past conversations, gating merges on CI evals, releasing behind a feature flag to shadow or canary cohorts, comparing online-eval failure rates per intent before and after, and agreeing rollback criteria and an owner in advance. The company says its own tool judges every conversation against plain-language rules and opens Auto Improve pull requests with fixes tested by replaying affected conversations, but does no CI gating, so it recommends pairing it with Promptfoo, DeepEval, LangSmith or Braintrust for the merge gate. Enterprises get confidence before rolling out a change to a customer-facing AI agent by stacking five checks: replay the change against real past conversations including the ones that failed , gate the merge on CI evals, release behind a feature flag to a shadow or canary cohort, compare online-eval failure rates per intent before and after, and agree rollback criteria and an owner in advance. Agnost AI is best for the production half: it judges every conversation on the new version against plain-language rules, alerts when failure rates move, and its Auto Improve PRs come with the fix tested by replaying the affected conversations. Agnost does no CI gating, so pair it with Promptfoo or DeepEval open source or LangSmith or Braintrust for the gate. This applies to every kind of change: a prompt edit, a model swap, a new or changed tool, a retrieval update. Each can break behavior that worked yesterday, and none of them throws an error when it does. Disclosure: We make Agnost AI. Facts about other tools come from their own docs, checked on November 18, 2026. | Stage | What it catches | What it misses | Tools | |---|---|---|---| | 1. Replay on past conversations | Regressions on real failures and real healthy traffic | New kinds of user input; live tool behavior | Your eval harness; Agnost AI Auto Improve replays affected conversations for its own fixes | | 2. CI eval gate | Known failure cases coming back on every PR | Anything not in the dataset | Promptfoo, DeepEval, LangSmith, Braintrust | | 3. Shadow or canary behind a flag | Problems at real traffic volume, on a small cohort | Rare intents the cohort didn't hit | Your feature-flag system, LaunchDarkly | | 4. Online evals, before vs after | Failure-rate and frustration changes per intent on the new version | Failures you have no rule for | Agnost AI, LangSmith, Braintrust, Arize AX, Opik | | 5. Rollback criteria and sign-off | Slow, argued rollbacks | Nothing on its own; it makes the other four actionable | A written runbook | Your strongest test data is your own production traffic. Build a replay set from: Redact personal data, and stub or sandbox side-effecting tools so a replay never moves real money. Compare the baseline and the candidate case by case, not as one average score. Turn production failures into regression tests https://dev.to/blog/production-failures-regression-tests/ has the full replay method. Where Agnost AI fits. When an Agnost alert fires and Auto Improve is connected, it opens a pull request with a fix tested against past runs: it replays the affected conversations and shows the results in the PR for a person to review. The replay is attached to the change instead of being a separate project. Turn the replay set and your hand-written cases into a test suite that runs on every pull request touching prompts, model config, or tools. A failing eval blocks the merge. promptfoo eval returns exit code 100 when at least one test fails, or when the pass rate falls below PROMPTFOO PASS RATE THRESHOLD 100% by default . There is a promptfoo/promptfoo-action for GitHub Actions. deepeval test run test llm app.py fails the build when a metric falls below its threshold. Cases marked flaky=True warn instead of failing. @pytest.mark.langsmith and a Vitest/Jest integration; assertion errors fail the CI run and results sync to LangSmith. braintrustdata/eval-action runs evals on pull requests and comments the results on the PR. LLM evals in CI vs production https://dev.to/blog/llm-evals-ci-vs-production/ has a complete GitHub Actions gate with Promptfoo. Agnost AI does not do this part. It has no offline datasets, experiments, or CI gating. Its role is to feed the gate: every conversation an Agnost rule flags is a candidate regression case for your CI suite. Passing CI means the change handles the cases you thought of. Production finds the rest, so expose it gradually: Put prompts and model settings behind a flag so rollback is a config change, not a deploy. LaunchDarkly, for example, documents managing prompts and model settings outside application code, with targeting rules and gradual rollout. Tag every conversation with the version it ran on, so you can compare. Once real users hit the new version, judge their conversations and compare against the old version: Where Agnost AI fits. Agnost judges every conversation, with no sampling, against plain-language rules once it has been idle for 15 minutes, and marks each assistant message pass or fail. Group results by intent and by version to compare before and after, and set alerts https://docs.agnost.ai/using-alerts on the rules that matter so a regression reaches Slack or email. After an Auto Improve PR is merged, Agnost re-checks the alert to see whether the condition improved. Judging is batched, not real time, so keep hard safety checks in application code. Decide these before rollout, while nobody is defending the change: | Item | Example | |---|---| | Critical rule | Any confirmed wrong refund, data leak, or policy breach: roll back immediately | | Failure-rate guardrail | Failure rate on a critical rule above its pre-release baseline for the canary cohort | | Frustration guardrail | Frustration rate on a top-five intent up by an agreed margin over the trailing week | | Operational guardrail | Cost per conversation or p95 latency above an agreed limit | | Minimum evidence | Enough canary conversations per top intent before expanding | | Rollback owner | One named person who can flip the flag without a meeting | Who signs off. An engineering owner for eval and replay results. A product or support owner for user-facing behavior. A risk, legal, or compliance reviewer for anything touching money, regulated advice, or personal data. Store the sign-off with the pull request, the eval results, and the rollout plan, which is also what an auditor will ask for. | Your situation | Use | |---|---| | No CI evals yet | Promptfoo or DeepEval on 20 to 50 cases from real failures | | You want CI and production evals from one vendor | LangSmith or Braintrust | | Failures only show up across whole conversations | A conversation-level judge in production: Agnost AI, or LangSmith thread evaluators | | You want production failures turned into tested fixes | Agnost AI Auto Improve, with your CI gate still running on its PRs | | You need enterprise controls VPC, audit logs, SLAs | See enterprise AI agent monitoring platforms https://dev.to/blog/enterprise-ai-agent-monitoring-platforms/ | Agnost AI pricing: Free covers 1,000 events a month; Starter is $49 a month; Pro is $499 a month; Enterprise is custom and includes self-hosted VPC deployment and audit logs. They stack five checks: replay the change against real past conversations, including the ones that failed; gate the merge on CI evals; release behind a feature flag to a shadow or canary cohort; compare online-eval failure rates per intent before and after; and agree rollback criteria and a named owner in advance. Promptfoo, DeepEval, LangSmith or Braintrust cover the CI gate. Agnost AI covers production: it judges every conversation on the new version, alerts when failure rates move, and its Auto Improve PRs replay the affected conversations to test a fix. Run the new prompt against a replay set built from real production conversations, especially ones that failed, plus a sample of ones that went well so you catch regressions. Gate the pull request on that eval in CI, for example with promptfoo eval exit code 100 on any failure by default or deepeval test run. Then ship it to a small cohort behind a flag before everyone gets it. Treat a model swap like a code change: run your replay set on both models and compare case by case, watch cost and latency as well as quality, and roll out to a canary cohort first. Model swaps often change tone, refusal behavior, and tool-call formatting, so include conversations that use every tool and every policy-sensitive intent. A replay set is a collection of real past conversations, redacted, that you re-run against a candidate change. It should include confirmed production failures the regression cases and healthy conversations the controls . Side-effecting tools are stubbed or sandboxed so a replay never refunds a real order. Write them down before rollout: for example, the failure rate on any critical rule rises above its pre-release baseline, frustration rate on a top intent rises by an agreed margin, a single confirmed high-severity failure such as a wrong refund or data leak, or a cost or latency regression. Name the person who can roll back without a meeting. Usually an engineering owner for the eval results, a product or support owner for user-facing behavior, and a risk, legal, or compliance reviewer for changes touching regulated topics, money, or personal data. Keep the sign-off record with the pull request, the eval results, and the rollout plan. No. Agnost AI does not run offline datasets, experiments, or CI gates. It judges production conversations against plain-language rules and alerts on failure rates, which makes it the before-and-after check after release. Use Promptfoo, DeepEval, LangSmith, or Braintrust for the CI gate. In 2025 many teams shipped prompt changes after a handful of manual tests. By 2026, enterprise teams expect a CI eval gate, replay against real production conversations, and online evals on every conversation after release. Tools that turn production failures into tested pull requests, such as Agnost AI Auto Improve, also arrived, so a fix ships with replay results attached. Originally published on agnost.ai https://agnost.ai/blog/safely-roll-out-changes-customer-facing-ai-agent/ .