{"slug": "your-agent-said-it-right-did-the-caller-hear-it-live-call-testing-in-voicetest", "title": "Your Agent Said It Right. Did the Caller Hear It? Live Call Testing in Voicetest", "summary": "The Voicetest project added live-call testing that runs voice-agent test cases as real WebRTC calls, with a simulated caller speaking through TTS and listening through STT so no human is on the line. An observer transcribes the audio each side publishes, recording both the intended text and what the caller actually heard, and a word-level diff flags defects like a phone number read as a quantity. The tool judges each call on both text and audio metrics, catching TTS regressions that text-only simulations pass.", "body_md": "A voice agent can pass every text simulation and still fail on a real call. The LLM produces the right sentence, and then the speech pipeline mangles it: a phone number read as a quantity, a drug name the TTS voice can't pronounce, an appointment time that STT on the other end turns into a different time. Text simulations never see any of this, because they stop at what the model *meant* to say.\n\nVoicetest now runs test cases as live audio calls. The same simulated caller that drives text runs joins a real WebRTC call, speaks through TTS, and listens through STT. Nobody has to be on the line. Every turn ends up with two texts: what was intended, and what was heard.\n\nThis post covers how live calls work, what they catch, and how to run one.\n\nIn a cascade voice pipeline (STT → LLM → TTS), the LLM's output is not what reaches the caller. It is an input to TTS, and the caller hears whatever TTS produced. A defect at that boundary looks like this:\n\n|  | Text | \n|---|---|\n| **Intended** (`content` ) | If you need to reschedule, call us at 415-555-0123. | \n| **Heard** (`heard` ) | If you need to reschedule, call us at four hundred fifteen, five hundred fifty-five, one hundred twenty-three. | \n\nBoth lines say the same thing to an LLM judge reading text. Only one of them is usable by a person writing a number down.\n\nVoicetest records this by adding an **observer** to each call. The observer subscribes to the audio each side publishes and transcribes it, so every turn in the transcript carries:\n\n`content`` heard`\nThe runs view marks up each turn with a word-level diff of the two, so a mangled number or a dropped \"not\" stands out without re-reading the whole call.\n\nA finished live call is saved as a normal result, marked `source_kind=\"live\"`, next to simulated and imported results in the same run history. It is judged three ways:\n\nThe second pass is the point. A metric like *\"Agent gave the caller the callback number\"* passes on the intended text above, and may well pass on the heard text too, since a number was technically given. Write the metric the way the caller experiences it, *\"Agent read the callback number digit by digit,\"* and the two passes disagree:\n\nThe text metric passes, because the LLM wrote the digits. The audio metric fails, because the caller never heard them. That's the TTS regression a text-only suite lets through.\n\nVoicetest already had [audio evaluation](https://voicetest.dev/docs/features/#audio-evaluation). It takes a finished *text* simulation and round-trips the agent's lines through TTS and STT afterwards. That's cheap, and it catches the same class of pronunciation defect.\n\nLive calls run the whole conversation over audio instead. The agent responds to what it *heard* the caller say, not to clean text, so STT errors on the caller's side change the course of the call the way they would in production. If the agent mishears \"Tuesday\" as \"Thursday\" in turn 3, the rest of the conversation follows from that mistake, and the transcript shows where it happened.\n\nUse audio evaluation as a fast check across a whole suite. Use live calls for the test cases where hearing matters: numbers, names, dates, confirmations, and anything a caller has to act on.\n\nLive calls need the voice services: a LiveKit server, a faster-whisper STT server, and a Kokoro TTS server. `voicetest up` starts all of them in Docker along with the backend:\n\n```\nuv tool install voicetest\nvoicetest up\n```\n\nThen run a test case as a call. In the Web UI, each test in the Tests tab has a **Call** button. It starts the call and switches to the Runs tab, where the transcript streams in turn by turn and the result is saved when the call ends.\n\nFrom the CLI, point at an agent in the database and a test by name:\n\n```\nvoicetest call \\\n  --agent-id <agent-id> \\\n  --tests tests.json \\\n  --test \"Refill with callback number\"\n```\n\nOver REST, it's `POST /api/agents/{id}/calls/start` with a `test_id`. Leave out `test_id`, or use the **Talk** button on the agent page, and the call waits for a human caller to join from the browser instead. Those calls are saved as runs too, judged against the agent's global metrics, which is useful for checking a prompt change by ear before writing a test for it.\n\n`voicetest down` stops the services.\n\nThe whole stack can run on your machine. The voice services are already local, and the agent, the simulated caller, and the judge can use a local model through llama.cpp's `llama-server`:\n\n```\nbrew install llama.cpp\nllama-server -hf Qwen/Qwen3-8B-GGUF:Q4_K_M --port 8080 --reasoning-budget 0\n# .voicetest/settings.toml\n[models]\nagent = \"llamafile/qwen3-8b\"\nsimulator = \"llamafile/qwen3-8b\"\njudge = \"llamafile/qwen3-8b\"\n```\n\nThis is also how voicetest tests itself. Its CI starts the stack in Docker and runs real calls end to end against a local model, with no external keys. Nothing in the call path needs a cloud account.\n\nObservation works on the published audio, regardless of how each side produced it. Today that's the cascade pipeline. A native speech-to-speech agent, with no separate text step, would be observed the same way.\n\nTiming isn't measured yet. Latency to first audio, dead air, and barge-in handling are the next metrics, and they need the same observer that's already on the call. If you run voice agents in production and have a timing failure you want caught, open an issue: [github.com/voicetestdev/voicetest/issues](https://github.com/voicetestdev/voicetest/issues).\n\nVoicetest is open source under Apache 2.0. GitHub: [github.com/voicetestdev/voicetest](https://github.com/voicetestdev/voicetest)", "url": "https://wpnews.pro/news/your-agent-said-it-right-did-the-caller-hear-it-live-call-testing-in-voicetest", "canonical_source": "https://dev.to/pld/your-agent-said-it-right-did-the-caller-hear-it-live-call-testing-in-voicetest-2jg1", "published_at": "2026-11-12 05:37:02+00:00", "updated_at": "2026-11-12 05:51:12.026219+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "natural-language-processing", "ai-products"], "entities": ["Voicetest", "LiveKit", "faster-whisper", "Kokoro"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/your-agent-said-it-right-did-the-caller-hear-it-live-call-testing-in-voicetest", "markdown": "https://wpnews.pro/news/your-agent-said-it-right-did-the-caller-hear-it-live-call-testing-in-voicetest.md", "text": "https://wpnews.pro/news/your-agent-said-it-right-did-the-caller-hear-it-live-call-testing-in-voicetest.txt", "jsonld": "https://wpnews.pro/news/your-agent-said-it-right-did-the-caller-hear-it-live-call-testing-in-voicetest.jsonld"}}