# Voice Agent Architectures Explained > Source: > Published: 2026-07-08 16:14:28+00:00 Every voice agent does the same three things. It listens, it thinks, and it talks back. The interesting part isn't the what — it's the how. And the "how" splits into three very different architectures, each with its own trade-offs on latency, control, cost, and the one thing that quietly decides whether your agent is any good: accuracy. So before you wire anything together, it's worth understanding the three patterns you can choose from — and why the step everyone treats as plumbing is actually the ceiling on how smart your agent can be. Strip away the framework names and a voice agent is a loop. A person says something. The agent turns that audio into text (speech-to-text, or STT). A language model reads the text and decides what to say (the LLM). That response gets turned back into audio (text-to-speech, or TTS). Then it waits for the next thing the person says and does it all again — ideally in about a second, because anything slower feels like a bad phone connection. Here's the thing most architecture debates miss: the model that thinks is only ever reacting to what the first step heard. If the STT layer mishears "my account ends in 4-0-1-5" as "4-0-1-8," the LLM confidently helps with the wrong account. Get the input wrong and everything downstream is wrong too. Keep that in mind — it's the theme that runs through all three architectures below. This is the classic approach, sometimes called the cascading or chained architecture. You pick a best-in-class model for each stage and connect them: a streaming speech-to-text API feeds transcripts to an LLM, and the LLM's response goes to a text-to-speech engine. Turn detection decides when the user is done talking so the LLM knows when to respond. The upside is control. You choose the most accurate STT model, the LLM that best fits your use case, and the voice you want. You can tune each stage independently — swap the LLM without touching transcription, add domain vocabulary at the STT layer, change voices at the TTS layer. For teams that need a genuinely differentiated product, that granularity matters. The cost is plumbing. You're now integrating three providers, reconciling three sets of latency, and debugging across three surfaces. Every millisecond of end-to-end latency is the sum of each hop plus the handoffs between them. It's the most flexible architecture and the most work. The newer approach collapses the pipeline into one multimodal model that takes audio in and emits audio out, reasoning over speech directly. It's elegant, and it's simple to start with — one API, one round trip. But simplicity has a price, and it's control. When a single model does everything, you can't swap the "ears" for a more accurate transcription model, you can't inspect the transcript to see what it actually heard, and you can't tune turn-taking or entity recognition stage by stage. These models are also built primarily around conversational fluency rather than getting a spelled-out email address or a 16-digit card number exactly right. When the same black box handles listening and speaking, an input error is invisible until the agent says something wrong. Speech-to-speech is a great fit for casual, low-stakes conversation. It's a harder sell when your agent has to capture names, numbers, and identifiers correctly on the first try. There's a third option that's easy to miss because it looks like the simple one from the outside and behaves like the flexible one on the inside: a unified voice agent API that runs the full STT→LLM→TTS pipeline for you behind a single connection. AssemblyAI's [Voice Agent API](https://www.assemblyai.com/products/voice-agent-api) is one example. You connect to a single WebSocket, stream audio in, and get audio back. Under the hood it's still a pipeline — real speech-to-text, a real LLM, real text-to-speech — but you get one connection, one bill measured in hours instead of token math across three invoices, and one set of logs instead of three dashboards. It's a flat $4.50/hr for all three stages, with roughly one-second end-to-end latency, and it works natively with tools like Claude Code so you can go from docs to a working agent the same afternoon. The framing that matters here is "invisible infrastructure." You're not adopting an opinionated agent platform where every agent ends up sounding the same. You keep control over conversation design, tool calling, and turn timing — you just skip the plumbing. It's built on Universal-3.5 Pro Streaming, so the listening step is handled by a model designed for exactly the hard tokens that trip agents up. There's no single right answer — there's a right answer for your product. Here's how the three patterns compare on the dimensions that usually decide it. | Dimension | Chained STT→LLM→TTS | Speech-to-speech | Unified voice agent API | |---|---|---|---| | Control over each stage | Highest | Lowest | High | | Time to first working agent | Slowest (three integrations) | Fast | Fast | | Transcript transparency | Full | None | Full | | Billing & debugging surfaces | Three | One | One | | Entity accuracy on hard tokens | Depends on chosen STT | Weaker | Strong (Universal-3.5 Pro Streaming) | Strong (Universal-3.5 Pro Streaming) If you're a large team building a deeply custom product and you want to own every layer, the chained pipeline earns its complexity. If you're prototyping something casual, a speech-to-speech model gets you talking fast. And if you want production-grade accuracy and control without integrating three vendors, a unified API is the pragmatic middle path — which is why it's where most teams building their first serious voice product end up. Notice what every row in that table has in common: none of them can fix a transcription mistake after the fact. The LLM can only reason over the words it was handed. So the accuracy of the speech-to-text layer sets the ceiling for the entire agent, no matter which architecture you pick. That's why the input model matters more than its "plumbing" reputation suggests. [Universal-3.5 Pro Streaming](https://www.assemblyai.com/universal-3-pro-streaming) is built for the tokens agents get wrong most often — emails, names, phone numbers, account IDs — with sub-300-millisecond latency and native code-switching across 18 languages. It uses punctuation-based turn detection so the agent knows the difference between a pause and a finished thought. In a chained pipeline you'd choose it as your STT layer; in the Voice Agent API it's already there. Either way, it's doing the job that decides whether the rest of your stack has a chance. For a closer look at where DIY stacks tend to strain under real traffic, our write-up on [where voice agent stacks start showing their limits](https://www.assemblyai.com/blog/where-voice-agent-stacks-start-showing-their-limits) is a useful companion read. Architecture is a series of trade-offs, but they all bottom out in the same place. Control versus simplicity, three bills versus one, transparency versus a black box — those are real decisions. What isn't up for debate is that the model doing the listening determines how good everything after it can be. Pick the architecture that fits your team, then give it the best possible ears. That's the part your customers will actually hear. Want the hands-on version? Our guide to [building with the Voice Agent API](https://www.assemblyai.com/blog/how-to-build-with-voice-agent-api) walks through a working agent, and [what are AI voice agents](https://www.assemblyai.com/blog/ai-voice-agents) covers the fundamentals if you're just getting oriented. A voice agent architecture is the way you connect the three core steps of a spoken conversation: speech-to-text (listening), a language model (thinking), and text-to-speech (speaking). The three common patterns are a chained STT→LLM→TTS pipeline, a single speech-to-speech model, and a unified voice agent API that runs the full pipeline behind one connection. A cascading pipeline uses a separate, best-in-class model for each stage, giving you full control and a visible transcript but requiring you to integrate and debug three providers. A speech-to-speech model handles audio-in to audio-out in one model, which is simpler to start with but hides the transcript and gives you less control over per-stage accuracy and turn-taking. All three can reach roughly one-second end-to-end latency when tuned well. A single speech-to-speech model removes handoffs between stages, while a well-built unified API like AssemblyAI's Voice Agent API reaches about one second end-to-end because the pipeline is optimized as a whole rather than assembled from three separate vendors. The language model can only respond to the words the speech-to-text layer produces, so any transcription error propagates through the entire agent. This is why entity accuracy on emails, names, phone numbers, and account IDs is the ceiling on agent quality — and why models like Universal-3.5 Pro Streaming are built specifically for those hard tokens. The best choice depends on how much control you need. Teams that want production accuracy without integrating three providers often choose a unified voice agent API; AssemblyAI's Voice Agent API runs STT, an LLM, and TTS through one WebSocket at a flat $4.50/hr, built on Universal-3.5 Pro Streaming for high entity accuracy.