# Evaluating voice agent quality at scale > Source: > Published: 2026-09-18 00:00:00+00:00 **A note on data and privacy.** Results here come from a production voice AI system serving residential property-management companies. The participating management companies are anonymized. No call transcripts are reproduced; every call description used as an example is redacted or synthetic, chosen to illustrate a type of interaction rather than to identify any real person. We [changed how our voice agent identifies callers](https://engineering.myhoai.com/posts/improving-caller-identification-for-voice-ai-agents), and on a handful of replayed calls the new logic worked beautifully. Calls that had previously spiraled into “can you spell that for me” now resolved on the first try. That felt like success. It wasn’t evidence. Replaying calls you already know were painful, through logic you built specifically to fix them, selects for exactly the cases your fix handles. The real question was harder, and it’s the one this post is about: **once a change is live, how do you know it made things better across thousands of calls a week that nobody has time to listen to?** ## The metric that looks right and isn’t The obvious answer is a pass/fail flag. Did the agent find the account, yes or no? It’s cheap to compute, easy to chart, and it answers the wrong question. Consider two calls that both register as matched. In the first, the agent recognizes the caller immediately and moves on to what they needed. In the second, the caller repeats their name three times, spells out their street, sits through several rounds of “I’m still not finding your account,” and eventually gets through. Both are a “yes.” A binary metric scores them identically. But the second call is the exact experience we set out to eliminate, and a pass/fail flag is structurally blind to it: blind to turn count, to spelling requests, to repetition, to the caller’s mounting irritation. What we had improved wasn’t the *outcome* of identification. It was the *experience* of it. And you cannot measure an experience with a flag. Measuring it properly means reading the whole conversation and judging how much effort the caller had to spend, which is precisely the thing that doesn’t scale to thousands of calls a day. ## Grading calls with a second model So we stopped trying to sample calls for human review and instead scored all of them with a separate model. The setup is simple. The evaluator (we call it the CSAT AI internally) receives a completed call transcript and a detailed rubric describing what good and bad look like, and returns scores. Every call gets an overall satisfaction rating plus a set of per-dimension scores, because a call can succeed on one axis and fail on another in ways an average conceals. Three of those dimensions matter for this story: overall satisfaction, caller identification, and task completion. Ratings run on a seven-point scale. The design choice that mattered most was refusing to ask the model to “rate this call.” Vague instructions produce inconsistent scores, and inconsistent scores are worse than no scores, because they look like data. So each dimension is written as an explicit, countable procedure instead. For caller identification, the rubric asks: How many identification attempts occurred? How many failure phrases appeared in the transcript (“I’m not finding,” “I’m still unable to locate your property”)? Was the caller asked to spell anything? Those counts drive the score deterministically. A 7 requires the account found on the first attempt with no failure phrases. A 6 permits one confirmation. The middle of the scale degrades with attempts, repeated information, and spelling requests. A 1 is a call where the account was never found at all. Writing the rubric as something closer to a scoring language than a prompt is what makes judgments consistent across calls, and what lets a small team audit a large volume of results. Two other rules hold it steady. The evaluator only scores completed calls. And when a call falls between two levels, it always rounds **down**. An evaluator that rounds up is just a slower way of fooling yourself. **A question we get asked a lot:** why not just survey the callers? Ask them directly with a post-call prompt and you’d get real self-reported satisfaction rather than a proxy for it. We considered it and passed, for three reasons. Response rates on post-call IVR surveys are low and heavily self-selected toward people who feel strongly, which is fatal when your test population is a few dozen calls. Abandoned calls, plausibly the worst experiences anyone had, are structurally absent from any post-call instrument. And a single survey score can’t be attributed to a dimension, so it can’t tell you whether *identification* improved or something else did. That’s a trade, not a free lunch. Our scores measure what a trained reviewer would judge from a transcript, which is a genuinely different thing from what a caller would report. ## The canary We didn’t cut the whole system over. About 3% of calls were routed to the new matching logic and the rest stayed on the existing path, small enough that if the new code misbehaved in a way testing had missed, few callers would feel it and rollback was immediate. The other benefit is what makes the comparison work: two live populations, running at the same time, on the same traffic, scored by the same evaluator under identical conditions. We pulled a single day from both paths. 2,154 calls on the old one, 76 on the new. **Caller-identification satisfaction was the headline.** The share of calls scoring 6 or 7 (the ones a caller would plausibly describe as good) went from 28.4% to 68.4%. The median score moved from a 4 to a perfect 7. Even at 76 calls, the 95% confidence interval on that difference runs from +29 to +50 points, nowhere near zero. **The knock-on effect surprised us.** We changed identification, so we expected identification to move. But overall call satisfaction rose too, from 8.9% to 19.7% in the top two boxes, and task completion from 11.8% to 26.3%. The reading we settled on: a call is a fixed amount of the caller’s patience. Time not spent spelling a street name is time available for solving the actual problem. Get people identified faster and more of them get their question answered, which is the outcome the management companies buying the agent care about most. ## Checking the grader Everything above rests on the evaluator’s scores being trustworthy. If it doesn’t grade the way a person would, comparing its output across two paths proves nothing at all. Before the canary ever launched, we built the rubric the slow way: a reviewer listened to 53 real calls, one at a time, compared each against the evaluator’s score, and every time the two disagreed, traced it to something the rubric was getting wrong, fixed the rubric, and scored again. That process moved about a third of the scores, almost all of them downward, as the rubric got stricter. By the end, a human had signed off on the score for every one of those 53 calls. That tells you the rubric was calibrated, but not that it holds up on calls nobody tuned it against. So on the first day of the canary we ran a second, independent check: a reviewer scored 19 live calls the rubric had never seen. Exact agreement with the evaluator was 68%, agreement within a point was 89%, and the two never diverged by more than two points on a seven-point scale. Enough to trust it, and enough to surface two defects we hadn’t anticipated. **The evaluator was counting planned questions as failures.** Our agent often asks for name and address as two separate questions. That’s the designed flow. But the rubric counts each piece of information the caller supplies as an “attempt,” so the evaluator read one successful two-turn identification as two failed ones and marked it down. That single mechanism accounted for five of the six disagreements. Our first instinct was to shrug it off. The evaluator scores low, so our improvements must be *understated*: we’re being hard on ourselves, which is the safe direction to err. However, a grader that marks everyone down by the same amount is harmless: the gap between the two paths stays the same. The problem is that this one doesn’t mark everyone down equally. It penalizes calls with more turns. Picture two calls, both of which end with the caller correctly identified. The old path took four turns to get there; the new path took two. Our evaluator deducts for turns it shouldn’t be counting, so it penalizes the old call twice as hard as the new one, and the gap between them looks bigger than it really is. None of this changes the headline. The scoring quirk moves calls by a point or two; the difference we measured between the two paths was forty points. It muddies exactly how much better the new path is, not whether it’s better. **The rubric had no way to score a call with no right answer.** Two calls concerned properties the management company doesn’t serve. The agent behaved correctly; there was no account to find. The evaluator scored both a 1. A rubric written to score a process can’t score the absence of one: obvious in hindsight, and invisible to any amount of automated scoring. With both defects understood and the improvement large enough to survive them, the new matching path was promoted to carry all production traffic and the old lookup retired. ## What we’d tell someone working on a voice agent **Transcripts are a proxy, and they leak.** Audio distortion, awkward pacing, robotic delivery: these matter enormously on a phone call and don’t appear cleanly in text. We got at them indirectly, by listening to hundreds of calls, finding the textual fingerprints these problems leave behind (a garbled rendering of a spelled-out email address; a caller asking the agent to slow down), and teaching the evaluator to look for those. It works. It’s still a proxy, and a transcript-only judge will miss things a listener catches. **Don’t let the judge grade its own family.** Models can flatter output from their own lineage. Our voice agent runs on GPT-4o mini; the evaluator runs on Claude Sonnet 4.5. Different family entirely, so the judge is never assessing its own work. **Validate the grader, then keep validating it.** We checked ours against human review before trusting it with any decision, and that was the right call: it’s what caught both defects. However, it isn’t a one-time exercise: traffic shifts, models get updated underneath you, and a spot-check by a person is the only thing that catches an evaluator drifting. Treat it as a standing process and write it into the SOP, or it quietly becomes something nobody owns. ## Takeaways The specifics here are about phone calls, but the problem is general. Most things worth improving don’t resolve to pass or fail, and artificial intelligence is likely to bring many products that improve an existing experience. Did the support ticket get handled *well*? Was the summary *good*? Did the onboarding flow *feel* smooth? Teams reach for a boolean because it’s easy to compute, then optimize against it for a year and wonder why the numbers moved and nothing got better. Three things we’d carry into the next problem of this shape: **If the thing you improved is an experience, a binary metric cannot see it.** Our two calls both counted as successes. One of them was the exact failure we’d set out to eliminate. Before you pick a metric, ask what a bad-but-passing case looks like. If you can describe one easily, the metric is wrong. **Write the rubric as a procedure, not an instruction.** “Rate this call” produces numbers that look like data and aren’t. “Count the attempts, count the failure phrases, note whether spelling was requested” produces judgments that hold steady across thousands of cases and can be audited by a person in minutes. The work is in specifying what you’re counting. **Decide in advance which way you’d rather be wrong.** We round down on every tie. It costs us some credit we could have claimed, and it means that when the number moves we believe it. An evaluator that flatters you is worse than no evaluator, because you’ll act on it. *This work has been submitted to IAAI-27.*