cd /news/ai-agents/how-to-detect-user-frustration-in-ch… · home › topics › ai-agents › article
[ARTICLE · art-172380] src=dev.to ↗ pub= topic=ai-agents verified=true sentiment=· neutral

How to detect user frustration in chatbot conversations

Agnost AI published a hands-on guide for detecting user frustration in chatbot conversations by tracking behavior across turns rather than per-message sentiment. The guide catalogs signals including repetition and rephrasing, corrections, escalation requests, short or all-caps replies, rage prompts, polite give-ups and mid-task abandonment, and recommends combining cheap rules with an LLM judge that reads whole conversations plus outcome joins to catch abandonment. It argues that sentiment scoring misses frustration because polite users often rephrase, thank the bot and leave while every message scores neutral or positive.

recommendation

Layer rules for cheap signals with an LLM judge reading full conversations and outcome joins for abandonment, then alert when frustration rises for a given intent.

more in playbook: Running AI in production
by read6 min views1 publishedNov 19, 2026

To detect user frustration in chatbot conversations, watch what users do across turns, not just the tone of one message: they rephrase the same request, correct the bot ("no, I meant…"), ask for a human, send short or all-caps replies, rage-prompt, give up politely, or leave mid-task. Agnost AI is best for teams that want an LLM judge to read every whole production conversation, flag the frustrated user messages, and alert in Slack or email when frustration rises for an intent. Measure the cheap signals (length, caps, timing, keywords) with rules, and the ones that depend on meaning (corrections, polite give-ups, sarcasm) with a judge that sees the whole conversation. Per-message sentiment alone misses most of it.

This is the hands-on guide: signals, how to measure each, example rule wordings, and alerting. If you want a single score, see how to build a frustration index. If you want a tool, see the best tools to detect user frustration in chatbots.

Disclosure: We make Agnost AI. Facts about other tools come from their own docs, checked on November 18, 2026.

Signal What it looks like Measure it with Common false positive
Repetition and rephrasing "How do I export?" then "where is the export button" Embedding similarity between user turns, or a judge A follow-up asking for more detail
Corrections "No, I meant the annual plan." "That's not what I asked." Judge (needs the previous bot turn) Users refining a request the bot got right
Escalation requests "Can I talk to a person?" "Give me a support email." Keyword rule plus a judge for paraphrases Products where the bot is meant to hand off
Short or all-caps replies Two sentences, then "ok", then "NO" Character count per turn, caps ratio Mobile users, acronyms, one-word confirmations
Rage prompts "JUST DO WHAT I ASKED." Insults, repeated demands. Caps and keyword rules, then a judge Anger at the user's problem, not the bot
Polite give-ups "Ok thanks, I'll figure it out." after a wrong answer Judge (tone is positive, behavior is not) A real thanks after a good answer
Abandonment mid-task A multi-step task stops after one bot turn and never resumes Timestamps plus a task-success event Users who got their answer in one turn

Rage prompts have their own definition page: what are rage prompts? There are three ways to measure, and you need more than one.

Method Catches Misses Cost
Rules (length, caps ratio, keywords, timestamps, embedding similarity) Short replies, all caps, explicit "human" requests, exits, near-duplicate turns Corrections, polite give-ups, sarcasm, anything phrased in a new way Cheap; runs on every message
Per-message sentiment Insults, all-caps anger, openly negative turns Polite frustration; flags anger at the problem rather than the bot Cheap
LLM judge on the whole conversation Corrections, rephrasing, polite give-ups, sarcasm, frustration that builds over several turns Nothing it can't read: outcomes that happen outside the chat Per-conversation LLM cost; needs calibrating
Outcome joins Abandonment: the task-success event never arrived Users who succeeded but were annoyed Needs product events

Sentiment scores the tone of one message. A frustrated user often stays polite: they rephrase twice, write "ok thanks", and leave. Every message scores neutral or positive. The opposite also happens: a customer furious about a late parcel scores very negative even when the bot fixes it on the first reply. Frustration is about the bot failing the user, which you can only see by reading the bot's turn and the user's reaction together. Our sentiment tools guide covers where sentiment does help.

conversation_id in the SDK or agnost.session_id on an Write each rule about one behavior and say what doesn't count. These work in any LLM judge: an Agnost AI intent or rule, a LangSmith thread evaluator, or an Opik rule.

**User-side signals** (label the user's message; in Agnost these are intents):

**Bot-side causes** (pass or fail on each assistant message; in Agnost these are rules):

Pairing the two tells you both how often users are frustrated and which bot behavior caused it.

In Agnost AI, an alert sits on any intent or rule with a count threshold, lookback window, and schedule, and goes to Slack or email. With Auto Improve connected, a fired alert opens a pull request with a fix tested against past runs: it replays the affected conversations and shows the results in the PR for a person to review. Agnost re-checks the alert after merge.

UserFrustrationMetric that scores conversation threads from 0.0 to 1.0 with an LLM.

For pricing, open-source status, and more tools, see the frustration detection tools roundup. Look at behavior across turns, not the tone of one message. The signals are repetition and rephrasing, corrections such as "no, I meant...", requests for a human, short or all-caps replies, rage prompts, polite give-ups, and abandonment mid-task. Measure the cheap ones (length, caps, timing, keywords) with rules, and the ones that depend on meaning with an LLM judge that reads the whole conversation. Agnost AI does this on every production conversation and alerts in Slack or email when frustration rises.

Asking the same thing again in different words, correcting the bot, asking for a person, replies that get shorter or switch to all caps, angry or insulting messages, a polite "ok thanks, never mind" after a bad answer, and leaving partway through a task. One sign on its own is weak evidence; two or more in the same conversation usually means the bot is failing that user.

Only part of it. Sentiment scores the tone of one message, so it catches insults and all-caps anger but misses users who stay polite while they rephrase, correct the bot, and quietly leave. It also flags users who are angry about their problem rather than the bot. Use sentiment as one input next to behavioral signals and the outcome of the conversation.

Partly. Message length drops, all-caps ratio, keyword matches for "human" or "agent", time to exit after the last answer, and embedding similarity between user turns all work without an LLM. Corrections, polite give-ups, and sarcasm depend on what came before, so they need a classifier or an LLM judge that reads the conversation.

Keep each rule to one behavior and say what does not count. For example: "Flag a user message that repeats or rephrases a request the assistant already answered. Do not flag follow-up questions that ask for more detail." Read the flagged and unflagged conversations, then tighten the wording until you agree with most verdicts.

Alert on the share of conversations with a frustration signal, per intent, against a trailing baseline, not on single messages. Mark releases on the chart so a jump after a prompt or model change is easy to spot. Send the alert with links to example conversations so whoever gets it can read the evidence first.

Agnost AI labels frustration on user messages after judging whole production conversations and alerts in Slack or email. Amplitude Agent Analytics scores sessions on a User Friction signal, Opik ships a UserFrustrationMetric for threads, and LangSmith runs thread-level evaluators you write yourself. See our roundup of chatbot frustration detection tools for the full comparison.

Yes. In 2025 most teams tracked thumbs-down rates and per-message sentiment. By 2026 several tools judge the whole conversation instead: LangSmith thread evaluators, Opik's frustration metric for threads, Amplitude's session signals, and Agnost AI's conversation-level judge. That makes polite frustration, which sentiment misses, measurable at scale.

Originally published on agnost.ai.

── more in #ai-agents 4 stories · sorted by recency
── more on @agnost ai 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/how-to-detect-user-f…] indexed:0 read:6min 2026-11-19 · —