{"slug": "why-realtime-is-the-future-of-speech-to-text", "title": "Why Realtime Is the Future of Speech-to-Text", "summary": "AssemblyAI's Universal-3.5 Pro Realtime streaming speech-to-text model achieves a 6.99% pooled word error rate on Pipecat's open STT benchmark, outperforming competitors like Deepgram Flux (15.58%), ElevenLabs Scribe v2 (9.76%), and Google Chirp3 (9.04%). The company argues that realtime transcription has become the default for high-value voice AI applications, with latency low enough that a person on a phone call cannot tell a machine is in the loop.", "body_md": "For most of the last decade, transcription was something you did *after* the fact.\n\nYou recorded a call, dumped the file into a queue, waited a few minutes (or a few hours), and got back a block of text. That was the deal. Batch, async, post-hoc—whatever you want to call it, the audio was already over by the time the model saw it. And for a long time, that was fine, because that's all the technology could reliably do.\n\nRecently, the field has quietly crossed the line where the most interesting, highest-value speech-to-text work happens *while people are still talking*. Realtime isn't a niche feature you bolt on for a captions demo. It's becoming the default that everything else gets measured against.\n\nHere's the argument.\n\nLook at where the energy is going in Voice AI right now.\n\n[Voice agents](https://www.assemblyai.com/blog/ai-voice-agents) that answer the phone and actually resolve the issue. Agent-assist tools that whisper the next best action to a rep mid-call. Live captions that keep up with a fast talker. Ambient scribes that write the clinical note as the visit unfolds. [Real-time conversation intelligence](https://www.assemblyai.com/blog/real-time-conversation-intelligence) that flags a compliance risk or a churn signal *before* the call ends, not in a report the next morning.\n\nEvery one of those is realtime-first. Not realtime-optional—realtime-first. The value evaporates if you make the user wait.\n\nCompare that to the classic batch workloads: transcribing a podcast back catalog, indexing a media archive, generating subtitles for content that already exists. Those are real, they matter, and they're not going away. But notice the pattern—they're all about audio that's *already done*. The frontier, the stuff people are building companies around in 2026, is about audio that's *happening*.\n\nSo the question flips. It used to be \"when do I actually need realtime?\" Now it's closer to \"why am I making this a two-step process at all?\"\n\nFor years you could have fast transcription or accurate transcription in the streaming setting, but picking both felt like a trap. Realtime models were noticeably worse than their batch cousins. So teams defaulted to batch when quality mattered and only reached for streaming when they absolutely had to.\n\nThat tradeoff is mostly gone. And it went away because two curves crossed the usability threshold at roughly the same moment.\n\nThe first is latency. [Universal-3.5 Pro Realtime](https://www.assemblyai.com/blog/universal-3-5-pro-realtime), the current streaming flagship, returns partial transcripts in a few hundred milliseconds and finalizes in the same ballpark. The prior generation, Universal-3 Pro Streaming, already ran around 150ms P50 post-VAD, roughly 240ms at P90, with time-to-complete-transcript near 250ms. Those aren't \"good for realtime\" numbers. Those are numbers where a person on the other end of a phone call can't tell a machine is in the loop.\n\nThe second curve is accuracy. On Pipecat's open STT benchmark, run on real agent conversations, Universal-3.5 Pro Realtime posted a **6.99% pooled word error rate**. For context, that same benchmark put Deepgram Flux at 15.58%, ElevenLabs Scribe v2 at 9.76%, and Google Chirp3 at 9.04%. The entity numbers are where it really shows up: 15.31% entity error rate versus 50.50% for the next-closest streaming competitor on that run, and 3.55% on phone numbers.\n\n| Streaming model | Pooled word error rate |\n|---|---|\n| Universal-3.5 Pro Realtime | 6.99% |\n| Google Chirp3 | 9.04% |\n| ElevenLabs Scribe v2 | 9.76% |\n| Deepgram Flux | 15.58% |\n\nSit with that for a second. A *streaming* model, transcribing live, is now clearing accuracy bars that batch models were bragging about a couple of years ago.\n\nAnd it keeps getting smarter mid-conversation. Universal-3.5 Pro Realtime takes direction from the application through agent_context—you pass in what the agent just asked, so a mumbled reply, a spelled-out account ID, or a one-word \"yep\" resolves correctly instead of turning into garbage. Across a benchmark of 20,000 voice agent audio files, passing that context **cut WER by 10.2%**, with fabrications down 18.3% and hallucinations down 17.2%.\n\nThen there's turn-taking. Old systems decided you were done talking by waiting for silence, which is why they interrupted you every time you paused to think. The end-of-turn detection here reads tonality, pacing, and rhythm—not just a silence timer—and fires in about 300ms. That's the difference between a conversation and a walkie-talkie.\n\nWhen fast and accurate stop being a tradeoff, the reason to default to batch quietly disappears.\n\nIt may be easy to read the section above and conclude realtime is just batch with a lower latency setting. It's not. It's a different engineering problem, and pretending otherwise is how teams ship a demo that falls apart in production.\n\nBatch has a luxury realtime never gets: the whole recording. A batch model can look ahead, reconsider, and clean up before it hands you a single finished answer. A streaming model has to commit to words it can't take back while the sentence is still being spoken.\n\nThat creates hard problems that simply don't exist in async:\n\n**Turn detection.** Deciding *when a person is done* is its own discipline. Cut them off and the agent talks over them; wait too long and it feels laggy and dead. Getting this right is arguably harder than the transcription itself.\n\n**Partial versus final transcripts.** Streaming emits guesses (partials) that it later confirms or revises (finals). Your application has to be built around that flicker—rendering something useful now, correcting it gracefully a moment later—without making the UI jump around like a broken slot machine.\n\n**Live context and memory.** A batch model gets the full conversation for free. A streaming model has to *carry* context forward as it goes. Universal-3.5 Pro Realtime keeps a short rolling memory (Context Carryover, on by default) so it remembers what was said thirty seconds ago instead of treating every utterance as a cold start.\n\n**Diarization with revision.** Figuring out who's speaking, live, is brutal—especially with overlap and short back-and-forth turns. The approach that actually works is to label speakers in real time and then send a single correction that re-clusters the whole session within about half a second of the stream ending. We wrote up the mechanics in [streaming speaker diarization](https://www.assemblyai.com/blog/streaming-speaker-diarization), and it's a good illustration of the general principle: realtime often means being *provisionally right now* and *definitively right slightly later*.\n\nNone of this is a reason to avoid realtime. It's a reason to take it seriously as its own category—and to be skeptical of any provider that treats streaming as an afterthought to their batch product.\n\nHere's where it gets interesting for the people who sign the invoices.\n\nWe've spent years talking about transcription accuracy as the thing that drives business outcomes, and it does—we've written a whole piece on [the true cost of inaccurate transcription](https://www.assemblyai.com/blog/true-cost-of-inaccurate-transcription). But in realtime systems, latency starts behaving like its own outcome variable.\n\nThink about a voice agent taking an inbound call. Every extra beat of dead air after the caller stops talking is a beat where the caller wonders if the line dropped, gets annoyed, or hangs up. I'd frame it as an open question the whole industry should be measuring more rigorously: **does streaming latency move call conversion and hangup rates?** My strong suspicion, from what we see across customers, is yes—and that the effect is larger than most teams assume. A voice agent that responds in about a second feels like a conversation. One that responds in three feels like a hold queue. Same transcript, wildly different business result.\n\nThat reframes the whole evaluation. If you're only benchmarking word error rate, you're measuring half the system. In a live setting, *how fast the words arrive* and *how naturally the turns flow* are load-bearing for the metric you actually care about, whether that's containment rate, CSAT, or conversion. This is part of why we keep arguing that [word error rate alone is broken](https://www.assemblyai.com/blog/word-error-rate-is-broken) as a way to judge a model—it says nothing about whether the system feels alive.\n\nI'd lose credibility fast if I told you batch is dead. It isn't, and I'd steer you toward it in plenty of cases.\n\n| Realtime (streaming) | Batch (async) | |\n|---|---|---|\n| Best for | Interactive work — voice agents, agent assist, live captions, ambient scribes | Already-recorded audio — archives, media libraries, overnight analytics |\n| Model | Universal-3.5 Pro Realtime | Universal-3.5 Pro |\n| Pricing | $0.45/hr base, billed on session duration | $0.21/hr, billed on audio duration |\n| Key advantage | Acts on speech while the conversation is still happening | Look-ahead over the whole file; simpler and cheaper at volume |\n\nIf the audio is already recorded and nobody's waiting on the result, batch is the right tool. Transcribing an archive of ten thousand old calls. Subtitling a media library. Running analytics over a quarter of sales conversations overnight. There's no live interaction to preserve, so paying the complexity tax of streaming buys you nothing.\n\nCost is real, too. Batch [speech-to-text](https://www.assemblyai.com/blog/speech-to-text) with Universal-3.5 Pro runs $0.21/hr of audio, billed on audio duration. Streaming with Universal-3.5 Pro Realtime is $0.45/hr base, billed on session duration—the whole time the WebSocket is open. For huge volumes of already-finished audio, the async path is simpler and cheaper, full stop.\n\nAnd batch still gets that look-ahead luxury. When you can afford to wait, a model that sees the entire file can sometimes squeeze out accuracy that no streaming system will match on the hardest audio.\n\nSo the honest position isn't \"realtime replaces batch.\" It's \"realtime becomes the default for interactive work, and batch keeps owning the archive.\" Two tools, clearer boundaries.\n\nIf you buy the argument that realtime is where the field is heading, here's what I'd actually evaluate—because \"we support streaming\" on a pricing page tells you almost nothing.\n\nAsk about latency *and* accuracy together, on realistic audio. A model that's fast on clean read speech and falls apart on a noisy phone call in an accent isn't solving your problem. That's why we benchmark on real agent conversations, not lab recordings, and publish the numbers on our [benchmarks page](https://www.assemblyai.com/benchmarks).\n\nAsk how turn detection works. Silence-timer only? Or does it read tonality and pacing? This single choice determines whether your product feels human.\n\nAsk whether the model can take live context. Can you feed it the agent's current question or a set of key terms mid-stream? agent_context and keyterm prompting are the difference between a model that guesses at \"my ID is A-as-in-apple, 4, 4, 9\" and one that gets it right.\n\nAsk about diarization and whether it revises. Live speaker labels that never get corrected will drift. The revise-at-end-of-turn pattern is what keeps them honest.\n\nAnd ask about the production stuff—concurrency limits, EU data residency, a signable BAA for healthcare workloads. Universal-3.5 Pro Realtime runs with unlimited concurrency and no rate limits, which matters the day your traffic spikes and you don't want to file a ticket to serve it.\n\nFor the full picture of how we think about the live path, the [streaming speech-to-text product page](https://www.assemblyai.com/products/streaming-speech-to-text) is the place to start, and if you're weighing accuracy specifically, [how accurate is speech-to-text in 2026](https://www.assemblyai.com/blog/how-accurate-speech-to-text) digs into the numbers.\n\nHere's the forward-looking part, and it's the thing I've been circling the whole time.\n\nAs realtime becomes the default, the transcript itself stops being the deliverable. For decades the output of speech-to-text *was* a document—text you'd read, search, or file. But a realtime transcript isn't something a human reads. It's something *another system* consumes, instantly, to make a decision: route the call, surface the answer, flag the risk, generate the next turn.\n\nThat changes what \"good\" means. The winning realtime models won't just be the ones with the lowest word error rate. They'll be the ones that emit the most *machine-actionable* stream—context-aware partials, confident finals, live speaker turns, end-of-turn signals—in a shape the next system can act on without waiting.\n\nIn other words, speech-to-text is quietly turning from a transcription tool into a real-time interface between human speech and software. The transcript is becoming plumbing. And once you see it that way, batch-versus-realtime stops being a feature comparison and starts looking like the difference between reading history and participating in the present.\n\nThat's the future I'd bet on. The rest of the field is already building for it.\n\nNo—and anyone who says so is oversimplifying. Realtime is becoming the default for *interactive* workloads like voice agents, agent assist, live captions, and ambient scribes, where making a user wait kills the value. Batch still wins for already-recorded audio: archives, bulk media, and overnight analytics, where it's simpler and cheaper. Batch async runs $0.21/hr with Universal-3.5 Pro; streaming runs $0.45/hr base. Think two tools with clearer boundaries, not a replacement.\n\nThey're the same thing. \"Streaming\" describes the mechanism—audio flows over a WebSocket connection and the model returns partial and final transcripts as it listens. \"Realtime\" describes the outcome—results arrive fast enough to act on while the conversation is still happening. AssemblyAI's streaming API uses wss://streaming.assemblyai.com/v3/ws.\n\nFor a natural back-and-forth, you want transcripts arriving in a few hundred milliseconds and turn detection firing quickly enough that the system doesn't talk over people or leave dead air. Universal-3.5 Pro Realtime returns partials in a few hundred ms, and its end-of-turn detection reads tonality and pacing to fire in about 300ms rather than waiting on a silence timer. End to end, a well-built voice agent lands around one second.\n\nMuch less than you used to. On Pipecat's open STT benchmark of real agent conversations, Universal-3.5 Pro Realtime posted a 6.99% pooled word error rate—ahead of Deepgram Flux (15.58%), ElevenLabs Scribe v2 (9.76%), and Google Chirp3 (9.04%). Passing agent context cut WER by a further 10.2% across 20,000 files. Batch can still edge ahead on the hardest audio because it sees the whole file, but the gap has narrowed dramatically.\n\nVoice agents, live agent assist for contact center reps, real-time [conversation intelligence](https://www.assemblyai.com/blog/real-time-conversation-intelligence), live captioning, and ambient clinical scribes. The common thread: another system consumes the transcript instantly to make a decision, so post-call transcription wouldn't work.\n\nOpen a WebSocket to the streaming endpoint and start sending audio—you'll get partial and final transcripts back right away. The [streaming speech-to-text product page](https://www.assemblyai.com/products/streaming-speech-to-text) has the details.", "url": "https://wpnews.pro/news/why-realtime-is-the-future-of-speech-to-text", "canonical_source": "https://dev.to/martschweiger/why-realtime-is-the-future-of-speech-to-text-2bii", "published_at": "2026-07-22 14:55:30+00:00", "updated_at": "2026-07-22 15:01:34.749898+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "natural-language-processing", "ai-products", "ai-infrastructure"], "entities": ["AssemblyAI", "Universal-3.5 Pro Realtime", "Pipecat", "Deepgram Flux", "ElevenLabs Scribe v2", "Google Chirp3"], "alternates": {"html": "https://wpnews.pro/news/why-realtime-is-the-future-of-speech-to-text", "markdown": "https://wpnews.pro/news/why-realtime-is-the-future-of-speech-to-text.md", "text": "https://wpnews.pro/news/why-realtime-is-the-future-of-speech-to-text.txt", "jsonld": "https://wpnews.pro/news/why-realtime-is-the-future-of-speech-to-text.jsonld"}}