{"slug": "one-line-of-text-beat-upgrading-our-ai-phone-agent-from-scribe-v2-to-v3", "title": "One line of text beat upgrading our AI phone agent from Scribe v2 to v3", "summary": "ThunderPhone found that passing the voice agent's last line as context to ElevenLabs' Scribe v2 improved transcription accuracy by 7.1 percentage points on live calls, matching or beating an upgrade to Scribe v3, which ElevenLabs released a day earlier claiming 27% fewer word errors. In tests on 451 caller turns from 302 real calls across 16 businesses, the two models stayed within about 2 percentage points of each other during calls, while on post-call recordings Scribe v3 reached 86.5% accuracy with key terms versus 82.7% for Scribe v2. ThunderPhone also reported that on recordings Scribe v3 supplied words for 9 of 10 unintelligible clips versus 5 for Scribe v2, a risk since a blank transcript lets the agent ask callers to repeat; both versions cost $0.22 per hour during calls and $0.20 per hour for recordings.", "body_md": "[← All posts](https://thunderphone.com/blog)\n\n# One line of text beat upgrading our AI phone agent from Scribe v2 to v3\n\nThunderPhone's AI phone agents answer and make business calls using a mix of transcription models tuned for each agent, including ElevenLabs' Scribe v2 (the old model).\n\nElevenLabs released Scribe v3 (the new model) yesterday, and we tested swapping it in right away.\n\nElevenLabs reports 27% fewer word errors on tests of people reading aloud. We wanted to know whether that held up on real business calls.\n\n## The test\n\nWe used 451 caller turns from real customer calls. A person listened to each turn and wrote down what was said. We focused on difficult moments: spelled-out names, addresses, emails, phone numbers, and noisy or quiet audio, leaving out calls that might contain health information.\n\nWe called a transcript \"right for the call\" if our agent would understand the caller the same way as from the person's version. We tried audio alone, a list of key words in `keyterms` (accepted by both APIs), and sending the agent's last line as context.\n\nElevenLabs' [during-the-call transcription API](https://elevenlabs.io/docs/api-reference/speech-to-text/v-1-speech-to-text-realtime) accepts `previous_text` with the first piece of audio. We sent: `A phone call between an AI agent and a caller. The agent just said: \"...\"`, replacing the dots with the last thing the voice agent said.\n\n## During calls: the agent's last line beat upgrading\n\nDuring calls, the models stayed within about 2 percentage points of each other in every setup. Neither had a clear edge.\n\nSending the agent's last line improved the old model's accuracy by 7.1 percentage points during calls. It helped the new model too, by less.\n\n## Recordings: better accuracy, more guesses\n\nOn recordings transcribed after the call, the new model reached 86.5% accuracy with key words.\n\nThe old model reached 82.7% in the same setup. Unlike during calls, the new model had a clear edge here.\n\nA phone agent must transcribe while the caller waits, whereas full recordings are transcribed after the call finishes. The versions used during calls and on recordings are separate models, so a win on recordings doesn't mean a win during calls.\n\nPrice wasn't a factor: both versions cost $0.22 per hour during calls.\n\nFor recordings after the call, both cost $0.20 per hour.\n\nOn recordings with key words, the new model got 36 turns right that the old model missed.\n\nBut it also missed 19 turns the old model got right. Its gains mostly came from names, yes-or-no answers, everyday conversation, phone numbers, and emails. A common fix was a short answer that the old model turned into the wrong word or an invented phrase.\n\nThe catch showed up in garbled audio. We checked 10 clips where a person couldn't make out any words. During calls, neither model had a clear edge.\n\nOn recordings, though, the new model supplied words for 9 of those clips.\n\nThe old model did so for 5. A blank transcript lets the agent ask the caller to repeat instead of responding to words the model guessed.\n\nHere's one of those clips, where the old model left the transcript blank and the new model wrote \"Roll in 800.\":\n\n## Where both models struggled\n\nWe grouped the 451 turns by what callers said; a turn could fit more than one group. Both models did best on yes-or-no answers, ordinary conversation, addresses, and general numbers and dates.\n\nEmails, company names, and personal names were hardest for both models. During calls, the old model was slightly ahead on names, spelled-out letters, and phone numbers, though the gaps were too small to call.\n\nPhone numbers and emails each had only 12 turns, so don't read much into those results.\n\nThe useful change was giving the model the last thing the voice agent said. We already had that text.\n\n## How we tested\n\n- **Sample:** 451 caller turns from 302 real calls across 16 businesses (about 18% chosen at random, the rest leaning toward hard moments); calls that might contain health information were excluded.\n- **Setup:** One run per setup in November 2026, during calls and after the call, testing`keyterms` and`previous_text` .\n- **Grading:** A person listened to every turn and wrote down what was said; a transcript counts as right if the agent would understand the caller the same way.\n- **Garbled audio:** 10 turns a person confirmed nobody could make out.", "url": "https://wpnews.pro/news/one-line-of-text-beat-upgrading-our-ai-phone-agent-from-scribe-v2-to-v3", "canonical_source": "https://thunderphone.com/blog/scribe-v3-on-phone-calls", "published_at": "2026-11-11 00:00:00+00:00", "updated_at": "2026-11-20 01:44:36.644104+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-tools"], "entities": ["ThunderPhone", "ElevenLabs", "Scribe v2", "Scribe v3"], "also_reported_by": [], "takeaway": "Passing the agent's last spoken line as previous_text context lifted Scribe v2's live-call transcription accuracy by 7.1 percentage points, while Scribe v3's edge appeared only on post-call recordings (86.5% vs 82.7% with key terms).", "recommendation": {"advice": "Test transcription models on your own real call turns before upgrading, and send the agent's last line as previous_text context, which gained 7.1 points on live calls in this 451-turn test.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/one-line-of-text-beat-upgrading-our-ai-phone-agent-from-scribe-v2-to-v3", "markdown": "https://wpnews.pro/news/one-line-of-text-beat-upgrading-our-ai-phone-agent-from-scribe-v2-to-v3.md", "text": "https://wpnews.pro/news/one-line-of-text-beat-upgrading-our-ai-phone-agent-from-scribe-v2-to-v3.txt", "jsonld": "https://wpnews.pro/news/one-line-of-text-beat-upgrading-our-ai-phone-agent-from-scribe-v2-to-v3.jsonld"}}