{"slug": "measure-the-queue-not-the-bot", "title": "Measure the Queue, Not the Bot", "summary": "A customer support leader argues that AI support agents should be measured by queue-wide resolution time and satisfaction, not deflection rates, citing that deflection metrics can be gamed and don't reflect actual problem-solving. The author reports that after rolling out an AI agent to a queue of 5,000 tickets per month, average resolution time dropped from 24 hours to 10 hours while satisfaction stayed above 95%, and notes that industry voices including Automattic's CX lead and a Smart Customer Service column have echoed similar concerns.", "body_md": "# Measure the Queue, Not the Bot\n\nWhen we rolled out an AI support agent to a queue of around 5,000 tickets a month, we never built a deflection dashboard. Not because we had some strong opinion against it. We just couldn’t see what it would actually tell us.\n\nDeflection measures the conversations that never reach a human. It doesn’t tell you whether the customer’s problem was solved. Someone who gives up is not a successful deflection. They might be a refund next week, or another ticket with a completely different subject line.\n\nThis August, I noticed three very different parts of the industry making essentially the same point. Automattic’s CX lead argued that [deflection is not a support strategy](https://automattic.com/2026/08/31/customer-support-ai-deflection/). A vendor-side column in Smart Customer Service argued that [resolution rate should replace it](https://www.smartcustomerservice.com/Columns/Vendor-Views/Stop-Counting-Deflections-Why-Resolution-Rate-Is-the-Metric-AI-Customer-Service-Actually-Needs-175977.aspx). And [China’s consumer-protection coverage](https://global.chinadaily.com.cn/a/202608/19/WS6a8506aaa3106bc57421c3a1.html) quoted a legal expert saying AI support should be judged on resolution and escalation efficiency, with a regulator behind the idea.\n\nSo, problem solved? Not quite.\n\nThe problem is that a resolution rate can be gamed too.\n\nClose-on-silence is one common example: if a customer doesn’t reply within 72 hours, the ticket is marked as resolved. Then there are bot-confirmed resolutions, where the agent asks “did that help?” and treats the customer disappearing as a yes. And if reopened tickets are logged as new tickets, one failure can conveniently turn into two “successful” resolutions.\n\nBasically, any metric that only looks at the bot’s part of the queue can be made to look good by carefully choosing what counts.\n\nFor me, the answer is to not give the AI its own scoreboard at all.\n\nYou look at one queue and one set of numbers. AI-handled and human-handled tickets should be measured together, using average resolution time and customer satisfaction across the whole queue. If the AI said it had solved something when it hadn’t, the customer would come back, and that would show up in our numbers. There should be no separate AI success metric to hide behind.\n\nThat also gives you a much safer way to scale it. You start the agent on 10% of the queue and only increase its share when the overall numbers hold up. Six months later, it'll be covering the entire queue. In our case average resolution time had dropped from around 24 hours to around 10, while satisfaction stayed above 95%.\n\nAnd those numbers were for everyone, not just the tickets where the AI happened to perform well.\n\nI understand why deflection dashboards are attractive. They’re easy to set up, easy to explain, and, conveniently, they usually go up.\n\nBut I think the metric you use to judge a support team should be something the customer would recognize as a success too.\n\nNobody ever wrote to us to say, “Thanks for deflecting me.” :)", "url": "https://wpnews.pro/news/measure-the-queue-not-the-bot", "canonical_source": "https://amitkvint.com/writing/measure-the-queue-not-the-bot/", "published_at": "2026-09-02 07:20:14+00:00", "updated_at": "2026-09-02 07:52:27.022164+00:00", "lang": "en", "topics": ["ai-products", "ai-tools", "ai-ethics"], "entities": ["Automattic", "Smart Customer Service", "China Daily"], "alternates": {"html": "https://wpnews.pro/news/measure-the-queue-not-the-bot", "markdown": "https://wpnews.pro/news/measure-the-queue-not-the-bot.md", "text": "https://wpnews.pro/news/measure-the-queue-not-the-bot.txt", "jsonld": "https://wpnews.pro/news/measure-the-queue-not-the-bot.jsonld"}}