A voice agent can complete the task and still lose the customer. Here is how to test the spoken experience before it reaches a real call.
A production voice agent needs a listener with a test plan, not just a successful demo.
Your voice agent says the right thing. It found the order, confirmed the price, and routed the caller correctly. Then it says the customer’s name with the wrong stress, slips into a different regional accent after a reconnect, or treats an English product name inside an Arabic sentence as a language switch.
None of those failures shows up in a basic task-success metric. Yet each can make an otherwise capable assistant sound unreliable, unfamiliar, or careless. In voice products, that is not a polish problem. It is part of the product contract.
OpenAI’s recent guidance for multilingual GPT-Live agents makes the issue especially concrete: teams should decide what triggers a language switch, keep regional speaking style distinct from language, and test switches in both directions, including after a session reconnects. The important shift is simple. Voice quality needs a repeatable release gate, not a few encouraging demo calls.
This guide shows how to build that gate. It focuses on GPT-Live, but the harness works for any real-time voice stack. You will define the speech contract, build a compact test corpus, collect machine and human evidence, and stop risky releases without turning every conversation into a lab experiment.
Most teams start with the reasonable question: “Did the agent complete the task?” That catches tool errors, wrong answers, dropped turns, and routing failures. It does not catch a more subtle problem: the agent may be linguistically understandable while failing the social expectations of the caller.
Consider a pharmacy refill agent that serves Spanish and English callers. It needs to say a patient name, a dosage, a date, a store location, and a safety escalation. A test transcript can look perfect while the audio still has one of these failures:
These are not all model bugs. They can come from the system prompt, the voice selection, session state, text normalization, an upstream tool result, or the handoff between live speech and a reasoning model. That is exactly why a single naturalness score is too blunt. You need evidence that identifies the failing layer.
A prompt describes how you hope the agent will speak. A speech contract describes what must remain true when the conversation gets messy. Write it before you create test cases.
Keep the contract short enough that a product owner, a linguist, and an engineer can read it together. For each market, define:
Do not write “use a natural accent.” That is not testable. Write a behavior that can be heard. For example: “Answer in the caller’s latest clear full-language utterance. A short English brand name, number, or acknowledgment inside a Hindi sentence does not change the conversation language. Keep the selected regional speaking style consistent after reconnect.”
That wording gives QA a way to decide whether a call passed. It also gives the implementation team a stable boundary when voice behavior changes with a model update.
You do not need thousands of prompts to learn whether a release is safe. You need a compact corpus that contains the phrases most likely to damage trust or alter an action. Start with 40 to 80 cases per launch market. Add cases whenever production reveals a new failure.
Each case should contain more than a text string. Record the intent, language, regional expectation, source of truth, severity, and what “good” sounds like. Treat the corpus as product data, version it in the repository, and review its changes like code.
{ "id": "pt-br-order-017", "market": "pt-BR", "category": "name-and-number", "caller_audio": "fixtures/pt-br/order_017.wav", "expected_language": "pt-BR", "critical_terms": ["Ana Luísa", "R$ 1.049,90", "QR-4821"], "must_not_switch_for": ["OK", "QR-4821"], "expected_action": "confirm_order", "human_review": true, "severity": "blocker"}
Use your actual domain vocabulary. A travel assistant needs airport codes, local place names, loyalty tiers, passport dates, and currency. A developer support agent needs package names, version numbers, commands, URLs, and product terms. A healthcare or finance workflow needs a separate, more tightly controlled list for regulated language.
1. Names and local entities. Include employee names, customer names with permission, locations, brands, and borrowed words. Get native-speaker confirmation for the intended rendering. A spelling list is not enough when stress or phonemes matter.
2. Numbers that change an outcome. Dates, decimal separators, currencies, units, confirmation codes, phone numbers, and street numbers are small strings with large consequences. Test natural spoken forms, not only a text transcript.
3. Code switching. Mix the language pairs that your callers actually use. Test a full switch, a product name embedded in a sentence, a number-only reply, and a caller who changes their mind. The desired result may be different for each.
4. Regional and register control. Test the intended market variation, not an abstract language label. Add formal and informal situations if your product supports both. The goal is intelligibility and appropriateness, not imitation for its own sake.
5. Conversation disruption. Interrupt the agent during a critical phrase. Simulate a long , a tool call, background noise, a session reconnect, and an escalation. Many accent and state failures appear only after this kind of disruption.
6. Repair behavior. Deliberately feed an ambiguous name or a misunderstood amount. A good agent asks, confirms, or offers a safe fallback. It does not bluff its way through a critical term.
Make the test matrix visible: language switching, names, interruptions, and reconnects should be separate lanes.
For a real-time agent, one utterance is a system outcome. It reflects caller audio, turn detection, session instructions, context, tool results, model decisions, and audio output. Capture evidence at each boundary so a failure is diagnosable.
For every run, store the fixture ID, model and voice version, prompt version, session ID, language decision, tool calls, transcript, audio artifact, timing, and reviewer result. Avoid storing unnecessary personal audio. Your fixture corpus should use synthetic or consented recordings, and recordings from real customers need their own retention and access policy.
Then split evaluation into three layers:
Automatic checks help at the task layer. Compare tool calls to expected structured output. Use a constrained transcript comparison for required phrases. Check that an expected transfer happened. Measure time from caller speech ending to the first useful agent audio.
But do not promote transcription to judge of all spoken quality. A transcript can preserve the words while losing stress, rhythm, regional appropriateness, or an audible mispronunciation. OpenAI’s GPT-Live evaluation guidance makes the same practical distinction: automated checks can stabilize broad evaluation, while human reviewers remain useful for naturalness, prosody, pronunciation, and pacing.
Human review is expensive only when it is vague. Make it focused. A reviewer should hear a short clip, see the speech contract and expected terms, and answer a handful of concrete questions.
Use at least two reviewers for a new market or a high-impact flow. If they disagree, record the reason instead of averaging away the disagreement. It may reveal that your contract is unclear, that a term has multiple acceptable pronunciations, or that the audience has a real regional split.
For routine changes, review a risk-weighted sample. Every changed blocker case gets a listener. Low-risk greeting cases can be sampled. This turns human judgment into a safety valve, not a release bottleneck.
When a voice failure occurs, teams often fix it in a prompt and move on. That works until the next model version, new voice, or localization release erases the fix. A pronunciation ledger makes the decision durable.
For each important term, keep the approved written form, language and regional context, an audio reference when permitted, a phonetic cue or IPA when helpful, the relevant prompt or lexicon instruction, test IDs, and owner. Do not force every word into the ledger. Reserve it for terms that affect identity, money, safety, or trust.
In GPT-Live, this can translate into a precise session instruction. The official prompting guide recommends specifying important pronunciations and gives an IPA-based example. Keep that instruction close to the term record, then make the test fixture prove the instruction still works after a reconnect and a language switch.
const speechPolicy = [ "Speak Brazilian Portuguese unless the caller clearly switches languages.", "A short English brand name, number, or OK does not change the language.", "Say Ana Luísa as /ˈa.na luˈi.za/ in Portuguese.", "For order codes and prices, speak slowly and offer to repeat them.", "After reconnect, retain these language and pronunciation rules."].join("\n");
await startLiveSession({ voice: "selected-production-voice", developerMessage: speechPolicy, metadata: { promptVersion: "voice-policy-14" }});
The code is intentionally simple. The difficult work is choosing a stable policy, attaching it to the correct session lifecycle, and proving that it holds under disruption.
Do not let a dashboard full of averages decide whether a multilingual voice release ships. Define a few failure types that automatically stop the rollout. Examples include a wrong dosage, a wrong payment amount, a failed identity confirmation, an unsafe escalation phrase, or a failure to recognize a declared language switch in a critical flow.
Then define quality budgets for the rest. A minor prosody issue in a noncritical greeting may be acceptable in a small canary. Repeated mispronunciation of a high-visibility city name is not. The budget should be stricter in a new market, after a voice-model change, and for regulated or revenue-impacting calls.
A useful release receipt answers four questions:
Start with a narrow canary. Watch language-transfer rate, caller repeat requests, escalation rate, critical-term corrections, call abandonment around confirmations, and complaints tagged to voice quality. Compare each measure by market. An average can hide a serious local failure.
Release evidence should compare a candidate voice against a known baseline before real callers discover the difference.
Voice QA is not a one-time localization project. Run the relevant slice of the suite whenever you change the voice, live model, prompt, language policy, text normalization rules, tool response template, turn detector, session recovery behavior, or audio transport.
That sounds like a lot, but the suite can be fast. Run a small smoke set on every pull request, the full critical corpus on a release candidate, and a broader human-reviewed set before a new market launch. Keep a baseline audio artifact for sensitive cases. Reviewers can then compare a candidate to a known acceptable output instead of relying on memory.
The biggest trap is treating a model upgrade as only an intelligence upgrade. A new model can preserve the task result while changing the pace, accent behavior, interruption style, or phonetic choices that users hear. If your product speaks, those changes are user-facing behavior.
On day one, choose one language pair or market, one high-value call flow, and ten critical terms. Write the speech contract. On day two, record or synthesize 20 caller fixtures that include names, numbers, a code switch, an interruption, and a reconnect. On day three, add run artifacts and a lightweight reviewer form. On day four, classify failures into prompt, voice, data, session, or application layers. On day five, set your first blocker rules and run a small canary.
This does not create a perfect voice agent. It creates a feedback loop that finds the failures a text-first evaluation cannot see. That is the difference between hoping a voice sounds local and being able to prove that it behaves acceptably in the conversations that matter.
It is a repeatable process for testing whether a voice agent understands and speaks appropriately across languages, regions, names, numbers, code switching, interruptions, and reconnects. It combines task checks, audio checks, and targeted human review.
Create baseline cases with a stated regional expectation, then rerun them after long s, language changes, tool calls, interruptions, and session reconnects. Human reviewers should judge whether the regional style remains acceptable, while logs identify the model, voice, and prompt versions behind a failure.
No. Use phonetic cues or IPA for names and terms where a mistake affects identity, safety, money, or trust. Keep the instruction close to a test case and audio reference so it does not become an unexplained prompt fragment.
It can catch missing words and some critical-term errors, but it cannot reliably judge prosody, pace, regional appropriateness, or every pronunciation problem. Use it as one signal, then reserve human review for the spoken qualities that change a release decision.
Begin with 40 to 80 cases per launch market, with extra attention on high-risk tasks. Grow the corpus from real failures. A small, well-maintained set of dangerous phrases is more useful than a large collection of generic greetings.
Block when the agent misstates a critical amount, dosage, identity detail, safety instruction, or required language behavior in an important flow. For lower-risk quality issues, use a documented budget, a limited canary, and a rollback trigger.
The takeaway: a voice agent earns trust one audible detail at a time. Build a test harness that listens for the details your transcript cannot see.
Multilingual Voice Agent QA: The Test Harness That Catches Accent Drift Before Customers Do was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.