Conversational AI assistants are everywhere now. Customer support, internal tools, sales, onboarding, there is hardly a product category left that hasn’t bolted one on. And yet, for how widespread they have become, most of them are still falling significantly short of what they could be. The reasons vary, but one pattern I keep running into is consistent: teams are not measuring the right things. They ship the assistant, they watch a handful of numbers go up, and they convince themselves the product is working. Meanwhile, users are quietly having a terrible experience.
I ran into this firsthand with a startup I was consulting with. Their support assistant looked great on paper, ticket closure rate was high, escalations to human agents were low. The dashboard was green across the board. But their NPS scores told a completely different story. When we dug into why, the answer was hiding in metrics they had never thought to track: conversation length, number of back-and-forths, how long it was taking the assistant to actually resolve anything. The bot was closing tickets, but only after exhausting the user with five or six unnecessary exchanges. It lacked context, so it kept asking questions it should have already known the answers to.
That investigation changed how I think about measuring conversational AI. What follows are the metrics that in my opinion actually matter, the ones that tell you whether your assistant is genuinely working, not just technically responding.
NPS measures how likely users are to recommend your product after interacting with the assistant. It is collected via a simple post-conversation survey: “How likely are you to recommend us to a friend or colleague?” on a scale of 0 to 10. Scores of 9–10 are promoters, 7–8 are passives, and 0–6 are detractors. Your NPS is the percentage of promoters minus the percentage of detractors.
NPS is the single metric that ties your assistant’s performance to actual business outcome. A support bot that closes tickets but leaves users frustrated will show up in your NPS before it shows up anywhere else. It is also one of the few metrics that captures the full arc of the interaction, not just whether the assistant answered, but whether the experience was good enough to talk about.
The model detects when the conversation has naturally ended and triggers an NPS survey dialog in the UI. Segment responses by conversation type, user cohort, and assistant version so you can isolate what is driving scores up or down.
The following code is a simple demonstration of collecting user feedback based on the conversation’s status. Please avoid using it in production, as it is far from AI engineering best practices..
import gradio as grfrom openai import OpenAIfrom pydantic import BaseModelclient = OpenAI()class AssistantResponse(BaseModel): response: str conversation_ended: booldef chat(message: str, history: list) -> tuple: messages = [ {"role": "system", "content": "You are a helpful support assistant."} ] + history + [{"role": "user", "content": message}] completion = client.beta.chat.completions.parse( model="gpt-4o", messages=messages, response_format=AssistantResponse ) parsed = completion.choices[0].message.parsed show_nps = parsed.conversation_ended return parsed.response, gr.update(visible=show_nps)def submit_nps(score: int, history: list) -> dict: if score >= 9: category = "promoter" elif score >= 7: category = "passive" else: category = "detractor" nps_result = { "nps_score": score, "category": category, "conversation_length": len(history) } print(f"NPS recorded: {nps_result}") return gr.update(visible=False)with gr.Blocks() as demo: gr.Markdown("# Support Assistant") with gr.Column(visible=False) as nps_panel: gr.Markdown("### How likely are you to recommend our assistant?") nps_score = gr.Slider(minimum=0, maximum=10, step=1, label="Score (0-10)") submit_btn = gr.Button("Submit") chatbot = gr.Chatbot(type="messages") gr.ChatInterface( fn=chat, chatbot=chatbot, additional_outputs=[nps_panel] )demo.launch()
Unlike NPS, which you ask for directly, implicit satisfaction is inferred from user behavior during the conversation. Signals like whether the user rephrased the same question, expressed frustration, or dropped off mid-conversation all tell you something about how satisfied they were, without ever asking.
Not every user will fill in your NPS survey. Implicit signals give you satisfaction data on 100% of your conversations, not just the ones where someone bothered to rate you. It also helps you spot patterns across sessions, which flows consistently frustrate users, which ones resolve cleanly, and where your assistant is quietly losing people.
Run a nightly batch job over all conversations from the day. For each conversation, send the full transcript to a lightweight model with a carefully designed rubric. The rubric matters more than you’d think, a vague instruction like “was the user satisfied?” will return inconsistent scores. Be explicit: define what satisfied, neutral, and frustrated look like in terms of observable signals in the conversation. You can find the full batch job implementation at the end of this article.
This is the metric that connects your assistant directly to business outcomes. Conversion measures whether the assistant successfully moved the user toward a meaningful business goal, a completed purchase, a ticket closed without human escalation, a booking confirmed, a payment page reached. Containment rate is the support-specific version: what percentage of conversations were fully resolved by the assistant without involving a human agent.
This is the metric that exposes the gap between “the assistant responded” and “** the assistant was useful**.” A high containment rate with a low NPS, as we saw earlier, means your assistant is technically closing tickets but leaving users exhausted. Tracked together, conversion and containment give you the clearest picture of whether your assistant is actually delivering value or just occupying the space where value should be.
For conversion, instrument the key downstream actions, payment page visits, booking confirmations, form submissions, and tag every link your assistant sends to users with UTM parameters. Something as simple as ?utm_source=chatbot&utm_medium=assistant&utm_campaign=support on every outbound link makes all the difference when you are trying to attribute what the assistant is actually driving. Without it, those conversions are invisible in your analytics.
For containment and outcome classification, you do not need a separate pipeline. In the same nightly batch job you are already running for implicit satisfaction, extend the rubric to also ask the model whether the conversation ended in a conversion, a clean containment, an escalation, or an abandonment. One job, one model call per conversation, two metrics out. You can find the full implementation at the end of this article.
Average conversation length measures the number of turns, a turn being one user message and one assistant response, it takes to reach a resolution. Turns are more useful than time because they are independent of how fast the user types.
This was the metric that cracked open the problem I described in the intro. The ticket closure rate looked fine, but conversation length told a different story, users were going through five or six exchanges to resolve something that should have taken two. Long conversations are not always bad; some tasks are genuinely complex. But when average length is high on conversations that should be simple, it is almost always a signal that the assistant is missing context, asking questions it should already know the answers to, or failing to understand the user’s intent on the first pass.
Store each message in your database with a conversation ID, a role field (user or assistant), and a timestamp. From there, measuring average turn count per conversation is a simple query, count user messages grouped by conversation ID, then average across the day. If you are already using Metabase or a similar BI tool, this becomes a dashboard you can monitor without writing any additional code. Segment by intent category if possible, because a billing question and a complex technical issue should not share the same benchmark. The signal is in the per-intent average, not the overall one.
There is no universal number that defines a normal conversation length. It depends entirely on the category and the type of business you are running. What matters is that you track it consistently over time. After a while a pattern will emerge, and that pattern will either seem rational given the complexity of your use case, or it won’t. If it doesn’t, you investigate and adjust. If it does, that pattern becomes your baseline, and any meaningful deviation from it becomes the signal worth paying attention to.
If your assistant uses tools, querying a database, calling an API, searching a knowledge base, looking up a user’s order; this metric tracks how many tool calls are made per conversation on average.
Tool calls are one of the clearest indicators of how efficiently your assistant is operating. Too many tool calls on a simple request means the assistant is either poorly prompted, lacking the right context upfront, or making redundant calls it should not need to make. Each unnecessary tool call adds latency, increases cost, and quietly degrades the user experience. Like conversation length, this metric rarely tells you something is wrong on its own, but tracked alongside turn count and NPS, it becomes a powerful diagnostic. When we looked at this metric alongside conversation length at the startup I mentioned earlier, the two told the same story from different angles.
If you are already using LangSmith, you already have this. LangSmith traces every run and logs each tool call within it, so you can see the exact number of tool calls per conversation, which tools were called, and in what order, without any additional instrumentation. If you are not using LangSmith, log each tool call with a conversation ID and tool name, and run the same kind of daily aggregation query you are already using for conversation length. The same Metabase dashboard can surface average tool calls per conversation alongside turn count, which makes it easy to spot when the two are moving together, a reliable sign that your assistant is working harder than it should.
Average cost per conversation measures how much each session with your assistant costs in terms of API usage, input tokens, output tokens, and tool calls combined. It is usually expressed as a cost per conversation or cost per resolved conversation, the latter being more meaningful since it accounts for whether the session actually delivered value.
This is the metric that keeps your unit economics honest. An assistant that costs $0.50 per conversation to run but only resolves 40% of tickets is effectively costing you $1.25 per resolution, and that changes the business case significantly. Tracking cost per conversation also surfaces inefficiencies that are invisible in quality metrics: unnecessarily long system prompts, redundant tool calls, or using a large expensive model for tasks a smaller one could handle just as well.
Every OpenAI API response includes a usage object with prompt_tokens and completion_tokens. Store both fields alongside the conversation ID and the model used in your messages table. From there, the cost calculation is straightforward: multiply token counts by the per-token price of the model, sum across all turns in a conversation, and you have your cost per session.
With those fields in your database, Metabase can do the rest. A simple calculated column — (prompt_tokens * input_price + completion_tokens * output_price) / 1,000,000 — gives you cost per turn, and averaging that grouped by conversation ID gives you cost per conversation. You can then cross-reference it with your resolution data from metric 3 to get cost per resolved conversation, which is the number that actually matters for your unit economics.
If you are using multiple models, a larger one for the assistant and a smaller one for the nightly batch jobs, track the model name per message so you can split costs cleanly in your dashboard.
Fallback rate measures how often your assistant fails to handle a request and falls back to a default response, something like “I’m not sure I understand, could you rephrase that?” or “I don’t have enough information to help with that.” It is the percentage of conversations that contain at least one fallback response out of all conversations.
A fallback is a small failure. One or two in a complex conversation might be acceptable, but a high fallback rate across your sessions is a direct signal that your assistant is encountering requests it was not designed or trained to handle. It could mean your intent coverage is too narrow, your system prompt is too restrictive, or users are coming in with expectations your assistant simply cannot meet. Left unmonitored, a high fallback rate quietly erodes user trust, users who hit a wall once are less likely to try again and it will simply increase your churn rate.
The most reliable way is to have the model explicitly signal when it is falling back, rather than trying to detect it from the text of the response after the fact. Add a boolean field to your structured response, similar to the conversation_ended field from metric 1 — called is_fallback. When the assistant cannot handle a request confidently, it sets that field to true alongside whatever response it gives the user. Store that field per message in your database and the rest is a simple Metabase query: count conversations with at least one is_fallback = true message, divide by total conversations, track over time.
Segment fallbacks by the user’s message that triggered them. That is where the real value is, not the rate itself, but the list of inputs your assistant keeps failing on. That list is your next product backlog.
Retry rate measures how often a user rephrases or repeats the same question within a conversation after receiving an unsatisfactory response. It is distinct from fallback rate, a fallback is the assistant explicitly failing, a retry is the user implicitly signaling that the response did not land, even when the assistant thought it did.
Retries are one of the most honest signals you have. The user is not filling in a survey or triggering a fallback, they are just trying again, which means they have not given up yet but they are not satisfied either. A high retry rate on a specific intent category usually means one of two things: the assistant is misunderstanding the request, or it is understanding it correctly but the response is not useful enough to act on. Those are two very different problems and they have different fixes, which is why tracking the retry rate alongside the actual retried messages is important.
This one is best detected in the nightly batch job. Add it to the rubric you are already using for implicit satisfaction and outcome classification, ask the model to flag whether the user repeated or rephrased the same request at any point in the conversation, and if so, how many times. Store the result as a retry_count field per conversation in your database. From there, Metabase gives you retry rate over time, average retries per conversation, and most usefully the conversations with the highest retry counts, which are the ones worth reading manually. You can find the full batch job implementation at the end of this article.
Drop-off point tracks where in a conversation users abandon the session without reaching a resolution. Unlike abandonment rate, which tells you that users left, drop-off point tells you where they left, which turn, which topic, which assistant response was the last thing they saw before they gave up.
Knowing that users are abandoning conversations is useful. Knowing exactly which assistant response is consistently the last one before they leave is actionable. A high drop-off on turn 2 across a specific intent category almost always points to a single broken response, fix that response and the drop-off moves. It is one of those metrics that sounds like a nice-to-have until you actually look at it, and then it becomes obvious why you should have been tracking it from day one.
Store a timestamp on every message. A conversation is considered abandoned when it ends without a resolution signal and the last message was from the assistant, meaning the user received a response and did not reply. In your nightly batch job, add a field to your rubric asking the model to identify whether the conversation was abandoned and what the assistant’s last message was before the user stopped responding. Cross-reference that with turn count to see at which point in the conversation the drop-off happened. In Metabase, grouping abandoned conversations by their last assistant message topic will surface the patterns quickly. You can find the full batch job implementation at the end of this article.
Return rate measures how often users come back to your assistant after their first session. It is the percentage of users who initiated more than one conversation over a given time window.
Return rate is a proxy for trust. A user who comes back has decided the assistant is worth trying again, which means their previous experience was good enough to not put them off. A low return rate is not always a problem; some assistants are designed for one-off interactions like booking a flight or filing a complaint. But for assistants built around ongoing engagement, like a support bot, an internal tool, or a product assistant, a low return rate is a quiet signal that the first experience did not leave users with enough confidence to come back. It is also one of the few metrics that captures long-term value rather than just session-level performance.
Store a user ID alongside every conversation in your database. Return rate is then a straightforward query: count users with more than one conversation divided by total unique users, over whatever time window makes sense for your product. Track it weekly rather than daily to smooth out noise, and segment by acquisition channel if you can — users who came in through different entry points often have very different return patterns.
Latency per response measures the time between a user sending a message and the assistant returning a response. It is usually tracked as an average and a p95 (the 95th percentile), because averages hide the tail cases that frustrate users the most.
Latency is one of the few metrics that affects perceived quality independently of actual quality. A correct answer that takes 8 seconds feels worse than a slightly less complete answer that arrives in 2. In conversational interfaces especially, where the expectation is something close to real-time, high latency breaks the flow of the interaction and signals to the user that something is wrong even when it isn’t. Track p95 in addition to average, your average might look fine while a significant portion of your users are waiting uncomfortably long on every message.
Store a sent_at timestamp on every user message and a responded_at timestamp on every assistant message. Latency per turn is the difference between the two. Average and p95 across all turns gives you your baseline. In Metabase, plotting this over time makes it easy to spot when a model update, a new tool, or an infrastructure change introduced a latency regression.
Token efficiency measures how many tokens your assistant consumes relative to the quality and length of its output. An assistant that produces verbose, padded responses is burning tokens and money without adding value. Track average output tokens per response alongside your resolution rate. If output tokens are climbing but resolution rate is flat or falling, your assistant is getting wordier without getting better. The fix is usually in the system prompt, tightening the instruction around response length and format.
Error rate tracks how often your assistant fails at the infrastructure level, API timeouts, failed tool calls, malformed responses, or any exception that prevents the assistant from returning a response at all. These are distinct from fallbacks, which are semantic failures. Error rate is operational. Even a small error rate compounds quickly at scale: a 2% error rate across 10,000 daily conversations means 200 users hitting a broken experience every day, and when you care enough about your users to not to turn them into numbers even a single failure matters. Log every exception with a conversation ID, error type, and timestamp. Track it in Metabase alongside fallback rate so you can distinguish between the assistant not knowing something and the assistant simply not working.
Correction rate measures how often users explicitly correct the assistant mid-conversation, telling it that its response was wrong, that it misunderstood, or that the information it provided was inaccurate. It is a strong signal of hallucination or knowledge gaps. Detect it in the nightly batch job by asking the model to flag any turn where the user corrected or contradicted the assistant’s previous response. A rising correction rate on a specific topic is usually a sign that your knowledge base or system prompt needs updating in that area. You can find the full batch job implementation below.
Override rate measures how often users ignore the assistant’s response and take a different action, asking to speak to a human, navigating away from a suggested link, or explicitly saying they will handle something themselves. It is the trust signal that is hardest to detect but most valuable when you do. A user who overrides the assistant has made a judgment call that the assistant cannot be trusted with this particular request. Detect it in the nightly batch job by looking for explicit override signals in the conversation, requests for human handoff, expressions of intent to act independently, or abrupt topic changes after a recommendation. You can find the full batch job implementation below.
All the metrics flagged for nightly detection throughout this article, implicit satisfaction, outcome classification, retry rate, drop-off point, correction rate, and override rate, can be computed in a single batch job. One model call per conversation, all metrics out.
The following code is a simple demonstration of collecting user feedback based on the conversation’s status. Please avoid using it in production, as it is far from AI engineering best practices. In production, instead of measuring all metrics in a single request, you may want to better engineer your context and divide the analysis into multiple separate requests. This is usually a tradeoff between cost and precision, there is no universal answer, and you should experiment on your own data, evaluate the results, and decide what works best for your use case. Additionally, the rubric included in the prompt below is not complete and is purely for demonstration purposes. In practice, rubrics vary significantly from business to business and should be carefully designed and validated against your specific data and goals.
from openai import OpenAIfrom pydantic import BaseModelfrom enum import Enumclient = OpenAI()class SatisfactionLevel(str, Enum): SATISFIED = "satisfied" NEUTRAL = "neutral" FRUSTRATED = "frustrated"class ConversationOutcome(str, Enum): CONVERTED = "converted" CONTAINED = "contained" ESCALATED = "escalated" ABANDONED = "abandoned"class NightlyAnalysis(BaseModel): # Metric 2: Implicit satisfaction satisfaction: SatisfactionLevel satisfaction_reason: str satisfaction_confidence: float # Metric 3: Outcome outcome: ConversationOutcome outcome_reason: str outcome_confidence: float # Metric 4: Conversation length intent_category: str # Metric 8: Retry rate retry_count: int retry_detected: bool # Metric 9: Drop-off abandoned: bool last_assistant_message_topic: str # Trust signals correction_detected: bool correction_count: int override_detected: bool override_count: intNIGHTLY_RUBRIC = """You are analyzing a support conversation. Return a structured analysiscovering the following dimensions:SATISFACTION: Classify overall user satisfaction as satisfied, neutral,or frustrated.- SATISFIED: Issue resolved, user confirmed positively, no repeated questions- NEUTRAL: Ambiguous outcome, no clear signal- FRUSTRATED: Repeated questions, explicit dissatisfaction, requested human, abrupt exitOUTCOME: Classify the conversation outcome.- CONVERTED: User completed a target business action- CONTAINED: Issue resolved without human escalation- ESCALATED: Handed off to a human agent- ABANDONED: User left without resolution or escalationINTENT CATEGORY: Classify the user's primary intent as one of:billing, technical, account, shipping, general.RETRY: Did the user rephrase or repeat the same question after receivinga response? If yes, how many times?DROP-OFF: Did the user abandon the conversation without resolution?If yes, what was the topic of the last assistant message before they left?CORRECTIONS: Did the user explicitly correct the assistant at any point?If yes, how many times?OVERRIDES: Did the user ignore the assistant's suggestion and indicatethey would handle something themselves, request a human, or navigate away?If yes, how many times?Return a confidence score between 0.0 and 1.0 for satisfaction and outcome."""def run_nightly_analysis(conversation_id: str, history: list) -> NightlyAnalysis: transcript = "\n".join( f"{msg['role'].upper()}: {msg['content']}" for msg in history ) completion = client.beta.chat.completions.parse( model="gpt-4o-mini", messages=[ {"role": "system", "content": NIGHTLY_RUBRIC}, {"role": "user", "content": f"Conversation transcript:\n{transcript}"} ], response_format=NightlyAnalysis ) return completion.choices[0].message.parseddef run_nightly_batch(conversations: list[dict]) -> list[NightlyAnalysis]: results = [] for conv in conversations: result = run_nightly_analysis(conv["id"], conv["history"]) results.append(result) print(f"[{conv['id']}] " f"satisfaction={result.satisfaction} | " f"outcome={result.outcome} | " f"intent={result.intent_category} | " f"retries={result.retry_count} | " f"corrections={result.correction_count} | " f"overrides={result.override_count}") return resultsif __name__ == "__main__": conversations = [ { "id": "conv_001", "history": [ {"role": "user", "content": "I can't log into my account"}, {"role": "assistant", "content": "Try resetting your password."}, {"role": "user", "content": "I already tried that"}, {"role": "assistant", "content": "Let me look into your account."}, {"role": "user", "content": "Finally, it's working. Thanks"} ] }, { "id": "conv_002", "history": [ {"role": "user", "content": "Where is my refund?"}, {"role": "assistant", "content": "Refunds take 5-7 business days."}, {"role": "user", "content": "That's wrong, I was told 3 days"}, {"role": "assistant", "content": "I apologize for the confusion."}, {"role": "user", "content": "I'll just call support directly"} ] } ] results = run_nightly_batch(conversations)
These metrics will not all matter equally for every product. A one-off booking assistant has no use for return rate. A support bot with no tool integrations does not need to track tool call counts. Start with the metrics that map directly to your use case, NPS, implicit satisfaction, and conversion rate are a safe starting point for almost everyone, and add the diagnostic ones as you mature.
What matters most is that you have a clear picture of what your assistant is actually doing in production, not just what it does in your test suite. The gap between those two things is where most assistants quietly fail. These metrics are how you close it.
15 Metrics You Should Be Measuring If You Are Building a Conversational AI Assistant was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.