{"slug": "a-real-customer-support-workflow-with-agentic-ai-where-it-helped-and-where-it-t", "title": "\"A Real Customer Support Workflow with Agentic AI: Where It Helped and Where It Didn't\"", "summary": "A support team deployed an agentic AI workflow that classified ticket intent, pulled account and order data through read-only tools, retrieved knowledge-base articles, and drafted replies, keeping refunds and account changes under human approval. The agent cut first-response routing time and produced handoff summaries that human agents valued most, but it missed refund-policy exceptions while sounding confident, hallucinated order details when lookups returned nothing, misread sarcasm as positive sentiment, and looped on repeated tool calls. The team concluded that fluency is not correctness and that money, legal terms, and exceptions must stay under human control.", "body_md": "Most posts about an **agentic AI customer support workflow** show the demo: a ticket comes in, the agent resolves it, everyone applauds. This post covers what happens after the demo, over several weeks of real tickets.\n\nWe put an AI agent into a live support workflow. It handled some work better than we expected, and it failed in ways that were uncomfortable to debug. By the end you'll know:\n\n**Honesty note:** This is one workflow, one team, and one set of tickets. Treat the numbers as a data point, not a benchmark.\n\nYou don't need to build this yourself to follow along, but it helps to have:\n\nBefore adding an agent, we mapped what a human did on a typical ticket:\n\nWe gave the agent narrow, explicit tools for steps 1 to 4 and kept step 5 under strict rules.\n\n```\nCustomer message\n      │\n      ▼\n [1] Classify intent + urgency\n      │\n      ▼\n [2] Look up account / order (read-only tools)\n      │\n      ▼\n [3] Retrieve relevant knowledge-base articles\n      │\n      ▼\n [4] Draft reply + confidence + sources\n      │\n      ▼\n [5] Policy check ──► auto-send   (low risk, high confidence)\n                 └─► human review (everything else)\n```\n\nThe main design decision was to start with **read-only tools and human-approved replies**. The agent could look things up and draft, but it couldn't issue refunds or change accounts.\n\nClassifying intent (billing, bug, how-to, account access) and urgency was the clearest win. Agents are good at turning messy, emotional free text into structured fields.\n\n``` python\nfrom pydantic import BaseModel\nfrom typing import Literal\n\nclass TicketTriage(BaseModel):\n    intent: Literal[\"billing\", \"bug\", \"how_to\", \"account_access\", \"other\"]\n    urgency: Literal[\"low\", \"normal\", \"high\"]\n    sentiment: Literal[\"positive\", \"neutral\", \"negative\"]\n    needs_human: bool\n    reason: str\n```\n\nForcing structured output meant the routing logic stayed deterministic even though the classification was not.\n\n**Result:** first-response routing time dropped from `[X]` to `[Y]`.\n\nMuch of a human agent's time goes to switching tabs: billing system, order history, previous conversations. The agent did this in seconds and attached a summary to the ticket.\n\nFor how-to questions with a clear answer in the knowledge base (password resets, setup steps, plan limits), drafts needed little or no editing. Human agents moved from writing to reviewing.\n\n**Result:** `[X]%` of drafts were sent with no edits, and `[Y]%` needed minor edits.\n\nWhen a ticket had 15 back-and-forth messages, the agent produced a short handoff summary. Human agents told us this was the feature they'd miss most.\n\nThis section is the main reason for the post.\n\nThe agent answered refund questions fluently, including cases where our policy had exceptions. It followed the general rule and missed the exception, and the tone gave no hint that it was unsure.\n\n**Lesson:** Fluency isn't correctness. For anything involving money, legal terms, or exceptions, don't let the agent decide alone.\n\nWhen two articles disagreed, the agent picked one without telling us. The agent wasn't the problem here. Our documentation was out of date, and the agent exposed it.\n\nOn vague tickets, the agent sometimes called the same lookup tool repeatedly with slight variations, hoping for a better result. We saw this in the logs before we saw it in the bill.\n\nA frustrated customer who wrote \"great, another broken update\" was sometimes read as positive. Sarcasm and distress were the weakest part of classification, and these are the tickets where tone matters most.\n\nEarly on, the agent escalated almost everything, which defeated the purpose. After we tuned it, it under-escalated a few tickets that should have gone to a human. Getting this balance right took more iteration than any other part of the system.\n\nWhen an order lookup returned nothing, the agent sometimes filled the gap with plausible but made-up details instead of saying \"I couldn't find this.\"\n\nNone of these are fancy. Each one came from a specific failure above.\n\n``` python\nMAX_TOOL_CALLS = 5\n\ndef run_agent(ticket):\n    calls = 0\n    while not done:\n        if calls >= MAX_TOOL_CALLS:\n            return escalate(ticket, reason=\"tool_call_limit_reached\")\n        result = agent.step(ticket)\n        calls += result.tool_calls\n```\n\nThese are plain code, not model judgment:\n\n```\nALWAYS_ESCALATE_KEYWORDS = {\n    \"refund\", \"chargeback\", \"legal\", \"lawyer\", \"cancel my account\",\n    \"data deletion\", \"gdpr\", \"security breach\",\n}\n\ndef must_escalate(ticket, triage):\n    text = ticket.body.lower()\n    return (\n        any(k in text for k in ALWAYS_ESCALATE_KEYWORDS)\n        or triage.urgency == \"high\"\n        or triage.sentiment == \"negative\"\n        or triage.needs_human\n    )\n```\n\nEvery draft had to cite the knowledge-base articles it used. No citation meant no auto-send:\n\n``` python\ndef can_auto_send(draft, triage):\n    return (\n        len(draft.sources) > 0\n        and draft.confidence >= 0.85\n        and triage.intent in {\"how_to\"}\n        and not must_escalate(draft.ticket, triage)\n    )\n```\n\nWe added an explicit instruction and a test for it:\n\n```\nIf a tool returns no result, say so plainly and escalate.\nNever infer order details, dates, or amounts that were not returned by a tool.\n```\n\nAuto-send was allowed only for low-risk how-to questions. Everything else got human review.\n\nWe didn't trust vibes. Before expanding the agent's scope, we replayed historical tickets through the workflow and compared the results against what a human had actually done.\n\n``` python\ndef evaluate(tickets):\n    results = {\"correct_route\": 0, \"safe_draft\": 0, \"bad_escalation\": 0}\n    for t in tickets:\n        out = run_agent(t)\n        results[\"correct_route\"] += out.intent == t.human_label.intent\n        results[\"safe_draft\"] += not contains_unsupported_claims(out.draft, out.sources)\n        results[\"bad_escalation\"] += out.escalated != t.human_label.should_escalate\n    return {k: v / len(tickets) for k, v in results.items()}\n```\n\nThe metric we watched most closely was **unsupported claims**, meaning statements in a draft that weren't backed by a source or tool result. Speed is easy to measure, but unsupported claims are what damage customer trust.\n\n| Metric | Before agent | After agent | \n|---|---|---|\n| Median first-response time | `[X]` | `[Y]` | \n| Tickets handled per human agent/day | `[X]` | `[Y]` | \n| Drafts sent with no edits | n/a | `[X]%` | \n| Unsupported-claim rate in drafts | n/a | `[X]%` | \n\nAgentic AI helped most where the work was repetitive, structured and low-risk: triage, context gathering, summarization and drafting answers to documented questions. It struggled with policy exceptions, emotional nuance, missing data and knowing when to stop.\n\nThe agent was useful because we assumed it would fail and built for that. If you're planning a similar workflow, start with read-only tools, put escalation rules in code, require sources, and review failures as carefully as you review wins.\n\n**Your turn:** Have you put an AI agent into a real support workflow? Where did it surprise you, for better or worse? Share your failure modes in the comments.", "url": "https://wpnews.pro/news/a-real-customer-support-workflow-with-agentic-ai-where-it-helped-and-where-it-t", "canonical_source": "https://dev.to/botsailorofficial/a-real-customer-support-workflow-with-agentic-ai-where-it-helped-and-where-it-didnt-1n70", "published_at": "2026-10-22 06:03:12+00:00", "updated_at": "2026-10-22 06:10:46.409635+00:00", "lang": "en", "topics": ["ai-agents", "ai-products", "large-language-models", "ai-tools"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-real-customer-support-workflow-with-agentic-ai-where-it-helped-and-where-it-t", "markdown": "https://wpnews.pro/news/a-real-customer-support-workflow-with-agentic-ai-where-it-helped-and-where-it-t.md", "text": "https://wpnews.pro/news/a-real-customer-support-workflow-with-agentic-ai-where-it-helped-and-where-it-t.txt", "jsonld": "https://wpnews.pro/news/a-real-customer-support-workflow-with-agentic-ai-where-it-helped-and-where-it-t.jsonld"}}