# How to test an AI chatbot before and after launch, and which metrics to track (2026)

> Source: <https://dev.to/shubhampalriwala/how-to-test-an-ai-chatbot-before-and-after-launch-and-which-metrics-to-track-2026-og8>
> Published: 2026-11-19 06:30:14+00:00

**Recommendation:** Build multi-turn test sets from real customer questions, simulate users with tools like Promptfoo or DeepEval, and gate every prompt or model change on regression evals in CI.
More in the playbook "Evals and testing for LLM apps": <https://wpnews.pro/playbooks/evals-testing>

**Test an AI chatbot in two phases.** Before launch, build a test set from real customer questions, run scripted and simulated multi-turn conversations, gate every prompt or model change on regression evals in CI, red-team it, and load-test latency. After launch, judge whole production conversations for failures and frustration, alert when a failure type spikes, and turn every confirmed failure into a new test case. Track test pass rate, multi-turn goal completion and p95 latency before launch, and verified resolution, escalation, silent-failure and frustration rates after.

Most "chatbot performance testing" advice stops at launch. That is the half that catches the failures you already imagined. The failures that cost you customers are the ones real users find in week two, so the plan below treats production as the second test environment and feeds what it finds back into the first.

*Disclosure: We make Agnost AI, which covers the post-launch half only. Facts about every other tool come from that vendor's own docs, pricing pages or repository, checked on November 18, 2026. Confirm details with each vendor.*

**Updated for 2026.** What changed since 2025: simulated-user testing is now built into the common tools (Promptfoo's `promptfoo:simulated-user` provider, DeepEval's `ConversationSimulator`, LangSmith's OpenEvals simulation), Promptfoo is now part of OpenAI and stays MIT-licensed, and production evaluators that judge a whole thread or group of traces are now documented by LangSmith and Braintrust, among others.

Your own guesses about what users will ask are the least useful test data you have. Pull questions from support tickets, chat logs from the bot you are replacing, sales call notes and search logs. Aim for:

For each case, write the expected behavior as something a judge can check: "quotes the 30-day refund window and links the form", not "gives a good answer". Promptfoo takes test cases as YAML, JSON, JSONL or CSV; DeepEval calls them goldens.

Single question-and-answer tests miss most chatbot failures: forgetting what the user said two turns ago, asking for the same information twice, looping, or handing off without context.

**Scripted conversations** fix every user turn in advance. Use them for flows that must work exactly, such as identity checks before account changes. Promptfoo's `initialMessages` lets you start a test from a fixed mid-conversation state.

**Simulated users** let an LLM play the customer, with a goal and a persona, so the conversation can go wherever your bot takes it. Promptfoo's simulated user provider:

```
defaultTest:
  provider:
    id: 'promptfoo:simulated-user'
    config:
      maxTurns: 10

tests:
  - vars:
      instructions: 'You are mia_li_3668...'
```

The conversation stops at `maxTurns` (default 10) or when the simulated user emits `###STOP###`. Two caveats from Promptfoo's docs: the target must accept messages in OpenAI chat format, and simulated-user messages are generated by Promptfoo's hosted models unless you set `PROMPTFOO_DISABLE_REMOTE_GENERATION=true`.

DeepEval's equivalent, from its docs:

``` python
from deepeval.dataset import ConversationalGolden, Persona
from deepeval.simulator import ConversationSimulator
from deepeval.test_case import Turn

conversation_golden = ConversationalGolden(
    scenario="Andy Byron wants to purchase a VIP ticket to a cold play concert.",
    expected_outcome="Successful purchase of a ticket.",
    persona=Persona(characteristics="Andy Byron is the CEO of Astronomer."),
)

async def chatbot_callback(input):
    return Turn(role="assistant", content=f"Chatbot response to: {input}")

simulator = ConversationSimulator(model_callback=chatbot_callback)
conversational_test_cases = simulator.simulate(conversational_goldens=[conversation_golden])
```

Replace `chatbot_callback` with a call to your bot. Then score each transcript against `expected_outcome` with a multi-turn metric. Run several simulations per scenario: an LLM user varies between runs, so one pass proves little.

Every prompt edit, model swap or tool change can break something that used to work. Run the test set on every pull request and fail the build when it regresses.

With Promptfoo:

```
npx promptfoo@latest init --example getting-started   # scaffold a config
npx promptfoo@latest eval -c promptfooconfig.yaml     # run in CI
npx promptfoo@latest view                             # inspect results locally
```

`promptfoo eval` exits with code 100 when at least one test fails, or when the pass rate falls below `PROMPTFOO_PASS_RATE_THRESHOLD` (default 100%, meaning no failures allowed). That exit code fails the CI job on its own.

With DeepEval, write pytest-style tests and run:

```
deepeval test run test_llm_app.py
```

DeepEval's GitHub Actions example passes `OPENAI_API_KEY` for the LLM judge. With Braintrust, `braintrustdata/eval-action@v2` runs evals on pull requests and posts the results as a PR comment; on other CI systems, run `bt eval evals/ --no-input --json`.

For a full GitHub Actions workflow and how CI evals differ from production evals, see [LLM evals in CI vs production](https://dev.to/blog/llm-evals-ci-vs-production/).

Before real users arrive, try to make the bot do what it must not: reveal its system prompt, leak another customer's data, follow instructions injected through a document or tool result, promise a refund policy you don't have, or produce harmful content.

Promptfoo has a red-team mode:

```
npx promptfoo@latest redteam setup    # web UI to configure targets and plugins
npx promptfoo@latest redteam run      # generate and run attacks
npx promptfoo@latest redteam report   # view findings
```

Its docs list jailbreaks, prompt injection, RAG poisoning, harmful or biased content, and checks against your own policies, mapped to OWASP, NIST and EU frameworks. Add your business rules as custom policies; the generic attacks won't know your refund terms. Track findings by severity and retest after each fix.

"Performance testing" in the traditional sense still applies. Drive the bot at your expected peak of concurrent conversations, plus headroom, with realistic multi-turn scripts rather than single requests, because long conversations carry long context. Measure:

Use whatever load tool your team already runs. The LLM-specific part is the test data: replay real transcripts, and include the tool calls the bot makes, so you test the provider and tool rate limits you'll actually hit.

Write these down before testing, not after. A typical set: every high-risk intent passes, overall pass rate at or above your baseline, no open high-severity red-team findings, simulated-user goal completion at or above your baseline on the top intents, and p95 latency under your target at peak load.

Pre-launch tests catch the failures you thought of. Production shows you the rest: phrasing you never tested, multi-step requests that drift, tools that return something unexpected, and answers that are wrong but sound right.

A response can look fine on its own and still leave the customer stuck three turns later. Judge each finished conversation against plain-language rules that encode what the bot must and must not do: "resolve the request or say clearly it can't be done", "never quote a price not in the pricing tool's output", "hand off with a summary when the user asks for a human". Count containment and escalation from handoff events, and resolution from the judge. [Chatbot resolution rate vs containment rate](https://dev.to/blog/chatbot-resolution-rate-vs-containment-rate/) covers the formulas.

Look for the signals users give when the bot fails them: rephrasing the same question, "no, I meant...", asking for a human, all-caps, giving up mid-flow, and coming back about the same issue a day later. Also watch for failures users don't notice: confident wrong answers, claimed actions that never happened, and tool-call errors the bot glossed over. See [chatbot frustration detection tools](https://dev.to/blog/chatbot-frustration-detection-tools/) for how tools detect these.

Alert when a failure type jumps, especially right after a deploy. A fixed threshold on a rule's failure count over a lookback window is enough to start. Send it where the owning team already works.

This is the step that makes testing improve over time. For each confirmed production failure:

[How to turn production failures into regression tests](https://dev.to/blog/production-failures-regression-tests/) walks through freezing evidence and replaying baseline against candidate.

| Metric | Stage | Formula or definition | Source | 
|---|---|---|---|
| Test pass rate | Pre-launch, CI | Passing cases / total cases, per intent | Eval tool | 
| High-risk intent pass rate | Pre-launch, CI | Must be 100% to ship | Eval tool | 
| Simulated goal completion | Pre-launch | Simulated conversations that met `expected_outcome` / simulated conversations | Simulator plus judge | 
| Red-team findings | Pre-launch | Open findings by severity | Red-team tool | 
| p95 turn latency | Load test and production | 95th percentile, user message to full reply | Traces | 
| Time to first token | Load test and production | First streamed token after user message | Traces | 
| Tool-call error rate | Load test and production | Failed tool calls / tool calls, per tool | Traces | 
| Verified resolution rate | Production | Conversations a judge says met the goal, no handoff / eligible conversations | Conversation judge | 
| Containment rate | Production | Conversations with no human handoff / eligible conversations | Handoff events | 
| Escalation rate | Production | Handoffs / eligible conversations, plus share judged late or unnecessary | Handoff events plus judge | 
| Silent-failure rate | Production | Contained conversations judged unresolved / contained conversations | Conversation judge | 
| Frustration rate | Production | Conversations with frustration signals / conversations | Conversation judge | 
| Repeat contact | Production | Same user, same issue, within N days / resolved conversations | Joined event data | 
| CSAT | Production | Positive / rated, always with response rate | Survey | 

No primary source publishes industry benchmarks for these. Set targets from two weeks of your own traffic, per intent. For SLO-style targets and alerting, see [AI agent SLOs and KPIs for production](https://dev.to/blog/ai-agent-slos-kpis-production/). For behavioral metrics in more depth, see [how to measure AI chatbot performance](https://dev.to/blog/how-to-measure-ai-chatbot-performance/).

| Tool | Stage | Multi-turn testing | CI gating | Red-teaming | Production monitoring | Licence or entry price | 
|---|---|---|---|---|---|---|
| **Promptfoo** | Pre-launch | Simulated user provider | Yes, exit code 100 | Yes, `redteam` | Not described | MIT, open source | 
| **DeepEval** | Pre-launch | `ConversationSimulator` | Yes, `deepeval test run` | Not covered here | Via Confident AI | Apache-2.0, open source | 
| **Cyara Botium** | Pre-launch, contact center | End-to-end journeys | Not stated on the product page | Security, jailbreak, prompt injection | "Continuous testing and monitoring" | Pricing not public | 
| **botium-core** | Pre-launch | Conversation test library | Run it in any CI | No | No | MIT, open source | 
| **LangSmith** | Both | OpenEvals simulation | pytest, Vitest/Jest | Not covered here | Multi-turn online evaluators | Plus $39/seat/mo | 
| **Braintrust** | Both | Not covered here | `eval-action@v2` | Not covered here | Online scoring, span/trace/group | Pro $249/mo | 
| **Agnost AI** | Post-launch | Judges real conversations | No | No | Every conversation judged, alerts | Starter $49/mo | 

**What it is.** An open-source (MIT) CLI and library for evals and red-teaming. Tests are YAML; graders range from string checks to LLM rubrics. Its README says Promptfoo "is now part of OpenAI" and "remains open source and MIT licensed."

**Best for.** Teams that want config-driven regression tests and red-teaming in the same tool, gated in CI.

**Limits.** Its docs describe running evals and red-team scans, not monitoring live production traffic. The simulated user needs a target that accepts OpenAI chat format, and it uses Promptfoo's hosted models by default.

**What it is.** An open-source (Apache-2.0) Python framework for LLM evals with pytest-style tests, multi-turn metrics and a `ConversationSimulator`. Results can optionally go to Confident AI, its hosted platform.

**Best for.** Python teams that want chatbot tests to look like unit tests.

**Limits.** The LLM judge needs a model key (OpenAI in the docs' CI example). Online evals on production traces and spans live in Confident AI, a separate hosted product.

**What it is.** Cyara closed its acquisition of Botium on April 5, 2022, and sells it as Cyara Botium. Its product page lists functional testing (including intent and entity coverage), regression, performance and load, security, end-to-end, and LLM agent testing for hallucination, inaccuracy and bias, across chatbots, voicebots and IVR. Cyara's AI agent page adds simulated personas, jailbreak and prompt-injection tests, and policy-compliance scoring. The open-source **botium-core** library ("Botium is for testing conversational AI") is MIT-licensed, not archived, and released version 1.15.19 on November 9, 2026.

**Best for.** Contact-center teams testing bots across chat, voice and IVR, often next to Cyara's other contact-center testing.

**Limits.** Pricing is not public; you go through a demo. CI/CD integration isn't mentioned on the Botium product page. botium-core is a library, not the full commercial product.

**What it is.** LangChain's tracing and evaluation platform. For testing, it documents multi-turn simulation with the open-source OpenEvals package (`run_multiturn_simulation`, `create_llm_simulated_user`), with trajectory evaluators that run on the finished conversation, inside its pytest and Vitest/Jest integrations. In production, multi-turn online evaluators judge whole threads.

**Best for.** Teams on LangChain or LangGraph that want test experiments and production threads in one tool.

**Limits.** Multi-turn online evaluators wait for an idle period (10 minutes by default, 2 minutes minimum), are capped at 10 per workspace, and process at most 500 threads per run. Base traces are kept 14 days.

**Pricing.** Developer is free for 1 seat and 5,000 base traces. Plus is $39 per seat per month with 10,000 base traces.

**What it is.** An evaluation platform built around experiments, datasets and scorers. `braintrustdata/eval-action@v2` runs evals on pull requests and comments the results. Online scoring runs scorers on production spans, traces or groups of traces, with sampling.

**Best for.** Teams whose releases run on experiments, who want production logs to feed datasets.

**Limits.** Trace and group scorers wait 30 seconds of idle time by default, which is short for chat users who pause between turns. Scores are metered.

**Pricing.** Starter is free with 1 GB and 10,000 scores. Pro is $249 a month with 5 GB and 50,000 scores. Users are unlimited.

**What it is.** Monitoring for AI agents in production: chat agents, voice agents (via transcripts), copilots and MCP servers. It is OpenTelemetry-native, with Python and TypeScript SDKs, and works with the OpenAI Agents SDK, Vercel AI SDK, LangChain/LangGraph, Mastra, Pydantic AI, Google ADK and Agno. It groups traffic by user and session and judges whole conversations.

**Best for.** The post-launch half of this plan: steps 6 to 9. You write rules in plain language. Once a conversation has been idle for 15 minutes, an LLM judge reads it and marks each assistant message pass or fail against every rule, in batches, with no sampling. It also detects user frustration, intents, tool-call errors, loops and silent wrong answers, and alerts to Slack or email. With Auto Improve, a fired alert opens a pull request with a fix tested against the affected past conversations, for a person to review.

**Limits.** Agnost is not a pre-launch testing tool. It has no CI gating, no offline datasets or experiments, and no red-teaming. Verdicts are pass or fail, not scores. Judging is not real time. Pair it with Promptfoo, DeepEval, LangSmith or Braintrust for the pre-launch half.

**Pricing.** Free covers 1,000 events a month. Starter is $49 a month (10,000 events). Pro is $499 a month (1,000,000 events). Enterprise is custom.

| If you need… | Use | 
|---|---|
| Config-driven regression tests and red-teaming in CI, open source | Promptfoo | 
| Chatbot tests written as Python unit tests, with simulated users | DeepEval | 
| Testing across chat, voice and IVR in a contact center | Cyara Botium | 
| Test experiments and production thread evals in one tool, on LangChain | LangSmith | 
| Experiment-driven releases with production logs feeding datasets | Braintrust | 
| Every production conversation judged, frustration detected, and failures turned into reviewed PRs | Agnost AI, next to one of the above for CI | 

Most teams need one tool from the pre-launch rows and one from the post-launch rows. The link between them is step 9: production failures become CI test cases.

Before launch, build a test set from real customer questions, run scripted and simulated multi-turn conversations, gate every prompt or model change on regression evals in CI, red-team for prompt injection and data leaks, and load-test latency. After launch, judge whole production conversations for failures and frustration, alert on spikes, and turn every confirmed failure into a new test case so it can't come back.

Before launch: test pass rate, multi-turn goal completion in simulated conversations, red-team findings by severity, p95 turn latency and time to first token under load. After launch: verified resolution rate, containment and escalation rate, silent-failure rate, frustration rate, tool-call error rate, repeat contact and CSAT with its response rate. Set targets from your own baseline, per intent.

The term covers two things. Load and latency testing checks that the bot answers fast enough at peak traffic, measured as p95 turn latency and time to first token at a target number of concurrent conversations. Quality performance testing checks that answers are correct and that whole conversations reach the user's goal. You need both: a fast bot that gives wrong answers fails, and so does a correct one that times out.

Use a simulated user: an LLM that plays a customer with a goal and a persona, talks to your bot for several turns, and stops when the goal is met or a turn limit is hit. Promptfoo has a promptfoo:simulated-user provider with a maxTurns setting, DeepEval has a ConversationSimulator, and LangSmith documents run_multiturn_simulation from the open-source OpenEvals package. Then judge the whole transcript against the expected outcome.

With Promptfoo, run npx promptfoo@latest eval in a pull-request workflow; the eval command exits with code 100 when a test fails or the pass rate drops below PROMPTFOO_PASS_RATE_THRESHOLD, which fails the job. With DeepEval, write pytest-style tests and run deepeval test run test_llm_app.py. Braintrust has braintrustdata/eval-action@v2, which posts results as a PR comment.

For CI regression evals and red-teaming, Promptfoo (MIT) and DeepEval (Apache-2.0). For experiments with datasets, LangSmith or Braintrust, which also score production traffic. For enterprise contact-center bots across chat, voice and IVR, Cyara Botium; the open-source botium-core library is still released under MIT. For judging whole production conversations after launch and alerting on failures, Agnost AI, which does not do CI gating.

Yes, in two forms. Cyara acquired Botium in 2022 and sells it as Cyara Botium, with functional, regression, NLU, load, security and LLM agent testing for chatbots, voicebots and IVR; pricing is not public. The open-source botium-core library is MIT-licensed, not archived, and had a release (1.15.19) on November 9, 2026.

When monitoring flags a failed conversation, confirm it by reading it, copy the user's turns and the state the bot saw into a test case with the expected behavior written as an assertion or rubric, add it to the CI suite, and check the fix passes it before merging. Keep a held-out set of past failures you don't tune against, so you can tell whether a fix generalizes.

*Originally published on [agnost.ai](https://agnost.ai/blog/how-to-test-ai-chatbot/).*
