Every support team that asks us about a voice agent has already heard a demo. The demo is always good. A pleasant voice answers a clean question in a quiet room, and somebody in the room says the word deflection. Then the thing goes live on a real number, and within a week somebody pulls the recordings and finds that a third of callers are pressing zero or simply hanging up.
I have looked at enough of those recordings to be fairly confident about where the damage happens, and it is almost never where the team expects. The voice gets blamed, the accent gets blamed, and speed gets blamed most of all because it is the number on the dashboard. In the recordings, callers leave when the agent has been allowed to start something it cannot finish, and then cannot pass them to a person without losing everything they just said.
Average handling time will not show you this. Averages smooth over the exact thing you need to see. What you want is the turn number at which the call died, counted across a few hundred real calls, and once you plot that you tend to get two spikes rather than a slope.
The first spike is early, usually turn one or two. That is the caller deciding they have reached a machine and they would rather not. Some of that is unavoidable. Some of it is your greeting promising more than the agent can do, which makes the caller test it immediately with the hardest thing they have.
The second spike is the expensive one. It sits somewhere in the middle of the call, after the caller has explained their problem properly, given an account number, answered two clarifying questions, and then hit a wall. The agent cannot do the thing. It says something polite. The caller has now invested two minutes and is further from a human than when they started. That is the call that turns into a complaint, and it is entirely a design decision rather than a technology limit.
My opinion, and it is an opinion rather than a measurement, is that most voice agent projects fail at scoping and then spend their budget on engineering. The question teams usually ask is what the model can handle, and the answer is a lot, which is why it is the wrong question. The useful one is which calls have a definite end that the agent can reach on its own.
A call has a definite end when there is a system the agent can read from or write to, and a sentence it can say afterwards that closes the matter. Where is my order. Move my appointment. Am I still under warranty. Reset this. Those end.
Calls that do not end are the ones where the resolution lives in somebody's judgement: refunds outside policy, complaints, anything where the caller is upset about a decision rather than confused about a fact. An agent can take the details, and that is genuinely useful, but the moment it implies it might resolve the matter you have built the middle of call hangup on purpose.
A test I use when a team hands me their intent list: for each intent, write the exact final sentence the agent says when it goes well. If you cannot write that sentence without a conditional in it, the intent is not ready for the agent to own.
The handoff is treated as an edge case in most projects I have seen, and it is the single biggest driver of whether callers tolerate the agent at all. A caller who reaches a person in fifteen seconds, with the agent having already told that person the account number and the problem, has had a decent experience even though the automation technically failed. A caller who has to start again from the beginning has had a worse experience than if you had never deployed anything.
So I would build the transfer path first, before a single intent is wired up. Concretely that means three things exist on day one.
There is a cost argument here too, and it cuts against the usual business case. If you scope narrowly and transfer quickly, your containment rate will look modest. Somebody will be disappointed by that number. I would rather have a modest containment rate on calls that were genuinely finished than a high one that is mostly callers giving up, which is what a containment metric counts if you do not separate the two.
Support dashboards are built for human queues and the default metrics mislead you here. These are the ones I would put on the wall instead.
| Metric | Why it tells you something | What the usual metric hides |
|---|---|---|
| Hangups by turn number | Separates people who never wanted a machine from people the agent lost mid call | Average handling time averages the two together |
| Resolved without transfer, sampled and listened to | Confirms the call actually ended rather than the caller leaving | Containment counts an abandoned call as a success |
| Time from escape request to a human voice | This is the number angry callers remember | Queue time measured from call start, not from the handoff |
| Repeat contact within 48 hours on the same issue | Catches confident wrong answers that closed the call falsely | First contact resolution measured on the agent side only |
The repeat contact number is the one I would watch most closely in the first month. A voice agent that is wrong and definite will look excellent on every other metric for about two weeks, because the calls are short and they end. The cost arrives later, in the second call and in the trust the caller no longer extends to your support line.
Callers talk over agents constantly, especially when correcting a misheard account number. If the agent keeps talking while being interrupted, the call is effectively over even if nobody hangs up yet. Stopping instantly and discarding the rest of the planned sentence is table stakes, and it is worth testing explicitly rather than assuming your platform does it.
Order references, postcodes, spelled surnames. Recognition on these degrades in exactly the conditions real support calls happen in, which is a caller in a car or a kitchen. Where you can, verify against a system rather than asking the caller to repeat: a partial match against the account on record is faster and less annoying than a third attempt at the full string.
I keep coming back to how much weight sits in the first eight seconds. An agent that names what it can do, in plain words, and mentions that a person is available, gets a noticeably calmer conversation than one that opens with "how can I help you today". The open question invites the whole problem, including the parts the agent cannot take.
Pick the two highest volume calls that have a definite end. Build the transfer and the logging first. Run the agent on a share of traffic rather than the whole line, with the old routing one config change away. Listen to twenty calls yourself in week one, which nobody ever wants to do and which tells you more than the dashboard will for a month. Expand the intent list only after the hangup curve stops showing a bump in the middle.
The volume and cost side of the business case depends entirely on your call mix, so I will not put a number on it here. Working that out for a specific line is part of the audit below.
If you have a voice agent live and the recordings are not matching the dashboard, our free AI audit includes listening to a sample of real calls and marking where each one was lost, and how we build voice systems covers the architecture underneath. If you would rather talk it through first, get in touch. Suresh B is the Founder & CEO of JarvisBitz Tech, an AI and software engineering company helping businesses build and scale production AI systems and digital products.