A user says:
“I want to increase my credit card limit.”
Somewhere behind that sentence, your voice assistant has to decide, in a fraction of a second, whether to just talk — or to stop, verify who’s speaking, pull account data, check policy rules, and only then respond.
That single decision reveals a lot about where Voice AI is right now.
Not long ago, most people’s only experience with a “talking system” was a robotic IVR menu reciting “press 1 for billing” a world away from actual AI. Those systems weren’t Voice AI at all; they were rigid, tone-based menu trees with no understanding of language. Today, real Voice AI assistants can sound remarkably human, sense a shift in mood, fall silent when interrupted, and pick the conversation back up naturally.
The reason isn’t simply “better speech recognition” or “better text-to-speech.” The real shift is happening one level down in the software architecture itself.
Voice AI is moving from cascaded pipelines toward native multimodal, speech-to-speech models. But does that mean the old architecture is on its way out?
Not necessarily.
And the answer matters more than it looks. It affects latency, compliance, cost per minute, system control, and ultimately whether your product feels human or simply like a very fast IVR.
This article walks through both architectures, where each one wins, and why the smartest systems being built today may not be choosing one over the other — but combining both.
The Cascaded Architecture, also called the sequential architecture, is still one of the most widely used approaches for production Voice AI.
It handles voice through a pipeline of distinct components:
Speech → Text → LLM → Speech
Each stage performs a specific job.
The user’s audio stream is processed in small chunks. Voice Activity Detection (VAD) algorithms help determine when the user starts and stops speaking, and the audio is converted into text.
Examples include Deepgram, Whisper, and AssemblyAI.
The transcript, combined with conversation state and system instructions, is sent to the LLM. Depending on the application, the model can also access external knowledge through RAG or trigger external systems through Function Calling.
Examples include Claude, GPT-4o, and Gemini.
The LLM’s text response is converted into audio. To reduce latency, the system can stream partial text to the TTS layer instead of waiting for the complete response.
Examples include ElevenLabs, Cartesia, and PlayHT.
Simple on paper — but this simplicity is exactly where the architecture gets much of its strength.
Flexibility and modularity are major advantages. You can combine the STT provider, LLM, and TTS system that best fit your requirements, and replace one component without redesigning the entire architecture. In enterprise environments where technology and vendor decisions change over time, that flexibility matters.
Security and control are another major advantage. The text layer creates a natural checkpoint. Before the response reaches the TTS system, the application can inspect and filter it using prompt-injection detection, hallucination checks, sensitive-data masking, KVKK/GDPR-related safeguards, and other text-based guardrails.
Cost optimization is also easier. Organizations are not necessarily tied to one provider’s audio-token pricing. A cost-effective STT service can be combined with an open-source or self-hosted LLM such as Llama and a suitable TTS provider, giving the organization more control over cost per minute.
The main weakness is the latency stack.
Every stage introduces processing time. STT processing, LLM time-to-first-token, TTS generation, network transfer, buffering, and orchestration all contribute to the final response time.
In some implementations, the accumulated latency can reach the high hundreds of milliseconds or even around 800–2000 ms, depending heavily on the models, providers, network, and infrastructure.
That can be noticeable in a real-time conversation.
There is also an information bottleneck.
When speech is converted into text, some information contained in the original audio may be lost or reduced. Tone, intonation, emphasis, irony, whispering, background noise, breathing, hesitation, and other paralinguistic signals are not fully represented by a transcript.
The LLM therefore receives a representation of what the user said, but not necessarily the full richness of how it was said.
The TTS layer then has to generate the voice from the resulting text and infer an appropriate style or emotion.
That separation works well for many applications, but it also creates a limit on how naturally the system can interact.
Native multimodal architecture is designed to reduce some of the latency and information-loss limitations of cascaded systems.
The core idea is simple: text is no longer necessarily the central path of the real-time conversational loop.
Instead of:
Audio → Text → LLM → Text → Audio
the system can process audio more directly and generate audio responses directly. This is commonly described as Speech-to-Speech (S2S).
However, “native” does not mean that text disappears from the overall application.
Transcripts can still be used for logging, monitoring, search, analytics, security, auditing, and business workflows. Tool calls and enterprise integrations can also involve structured text or JSON.
The architectural difference is that text is no longer necessarily the only or primary representation used for the real-time conversational loop.
Two concepts are particularly important: audio tokenization and neural audio codecs, and shared multimodal representations.
Audio tokenization and neural audio codecs. Just as text-based LLMs represent language using text tokens, native audio systems can use learned audio representations or audio tokens to process continuous speech. Technologies such as EnCodec, SoundStream, and SpeechTokenizer are examples of neural audio codecs used in the broader audio-model ecosystem. On the output side, audio representations can be decoded into speech and streamed back to the user.
Shared multimodal representations. Instead of treating text and audio as completely separate layers, native multimodal models can process text, audio, images, and other modalities within a shared model architecture. This gives the model access to richer information about the interaction.
The model is not limited to a sequence of words.
Depending on the model and implementation, information such as voice characteristics, speaking speed, emphasis, accent, background noise, breathing s, stress, happiness, and hesitation can contribute to how the interaction is interpreted.
This can also influence how the model responds.
Instead of producing the same neutral voice in every situation, a native audio system can potentially adapt its intonation, speaking style, or emotional expression to the context.
This is a major part of why native multimodal systems can feel more natural.
One of the biggest attractions of native multimodal systems is responsiveness.
By reducing several intermediate transformation steps, these systems can target response times in the few-hundred-millisecond range, depending on the model, provider, network conditions, and implementation.
For example, OpenAI reported that GPT-4o could respond to audio input in as little as 232 milliseconds, with an average response time of around 320 milliseconds in its published measurements. That should not be interpreted as a universal latency standard for all native multimodal systems, but it demonstrates the level of responsiveness these architectures can target. The practical benefit is more important than the number itself.
The assistant can potentially respond more naturally when the user whispers, stop when interrupted, adapt to changes in speaking style, handle overlapping speech, and react to laughter or changes in tone.
Which brings us to one of the sharpest differences between the two architectures: turn-taking.
Knowing when someone has finished speaking is just as important as understanding what they said.
Cascaded systems typically rely on external VAD mechanisms based on signals such as silence duration, volume, and voice activity.
A system might treat a certain period of silence as the end of the user’s turn.
The problem is obvious: a user may simply to think.
If the system interprets that as the end of the sentence, it can start speaking too early. It works, but it can feel blunt.
Native S2S systems can combine acoustic signals with semantic signals when deciding whether the user has finished speaking.
Acoustic signals can include a drop in pitch, breathing patterns, s, and changes in sentence-final intonation.
Semantic signals can help determine whether the user’s thought is actually complete.
“The weather today is…”
is clearly unfinished.
“The weather today is very nice.”
is clearly complete.
By considering both channels, the system can make a better decision about whether the user is simply pausing or has actually finished.
This becomes particularly important when the user interrupts the assistant.
When the user starts speaking while the AI is talking, the system needs to stop its current audio output and return to listening mode.
Modern real-time voice systems can support this kind of interruption handling, making barge-in a key capability for natural conversational interfaces.
The difference is important.
Traditional IVR systems often force the user to wait for the system to finish.
A natural conversational system needs to understand that the user has taken the turn back.
This is where native S2S becomes genuinely challenging.
In a cascaded architecture, Function Calling is relatively predictable.
The LLM generates a structured request, such as JSON, the application executes the API or database operation, the result is returned to the model, and the final text response is sent to TTS.
Clean, sequential, and relatively easy to control.
In a native S2S system, however, the audio stream is continuous.
If the model needs to query a database or call an API during the conversation, the application has to decide what happens to the live audio while that operation is running. Go completely silent, and the user may assume the connection has dropped.
One possible solution is to generate human-like filler responses such as:
“I understand, let me check that for you.”
or:
“Hmm, let me take a look.”
These responses can bridge the gap while the backend operation is running.
But synchronizing filler audio, the API call, model reasoning, interruption handling, and real-time audio generation requires careful orchestration.
This is why native S2S can still present significant engineering challenges in business scenarios that require frequent API calls, deterministic data access, and strict enterprise integration.
Native multimodal systems bring several important challenges that can be easy to underestimate.
In a cascaded system, engineers can inspect the generated text before it reaches the user.
In a native audio system, that control point becomes less straightforward.
Guardrails may need to operate across multiple representations, debugging can become harder, and model behavior can feel more like a black box because the conversational loop is more tightly integrated.
Audio processing can consume tokens at rates measured in the tens of tokens per second, depending on the model and audio representation.
That means a one-minute voice interaction can generate thousands of audio tokens.
The exact token rate is not universal. It varies by model, codec, sampling configuration, and API.
But the architectural implication is clear: continuous audio can consume significantly more model context than a short text interaction.
Higher token consumption can cause context windows to fill faster than in text-based systems.
This matters for long telephone calls, complex service workflows, and long-running assistants.
As conversations become longer, applications may need mechanisms such as context compression, summarization, or pruning to preserve the most important information.
This is already becoming an explicit consideration in real-time multimodal systems.
The cost implication is equally important.
Audio input and output can be priced differently from text, depending on the provider and model. When a system processes large volumes of audio for long periods, both token consumption and pricing can become major architectural considerations.
For a large contact center, the difference can scale from a small optimization into a significant operational cost. The right question isn’t:
“Which architecture is newer?”
It’s:
“Which architecture matches what my product actually needs?”
If your priorities are strict compliance, deterministic responses, enterprise data integration, RAG, cost control, on-premise deployment, and easy debugging, **cascaded architecture may still be the stronger choice**.
If your priorities are very low latency, natural turn-taking, emotional interaction, and human-like conversational behavior, **native multimodal architecture becomes much more attractive**.
Despite the excitement around native models, there is no simple evidence that cascaded Voice AI is disappearing.
In regulated and enterprise environments, the reasons for using cascaded architectures remain strong.
Financial services, healthcare, insurance, and other regulated environments can require strong compliance controls, deterministic access to enterprise data, predictable workflows, clear auditability, and tight cost management.
That is exactly where the modular nature of cascaded architectures becomes valuable.
In banking, finance, healthcare, and other highly regulated environments, organizations may have strict requirements around where sensitive data can be processed.
KVKK and GDPR requirements can directly influence architecture decisions.
Cascaded systems can be attractive in these scenarios because their components are modular.
A lightweight STT model such as Whisper-tiny, an optimized local implementation, and a quantized local LLM such as Llama-3–8B-Instruct can provide a path toward localized or offline processing, depending on the specific hardware and software stack.
Native S2S models can require more capable hardware because audio and language are processed within a much more integrated model architecture.
Quantizing and deploying such models on mobile devices or low-cost local servers can therefore be challenging.
This does not mean native models cannot run locally.
It means that model size, GPU memory, inference performance, optimization, and available hardware become important architectural constraints.
Where requirements include no internet dependency, strict local processing, strong data privacy, or limited hardware budgets, cascaded architecture can remain a highly practical option.
Native multimodal architecture is particularly attractive for use cases where natural interaction is itself a major part of the product value.
Examples include language-learning applications, gaming NPCs, personal coaches, empathetic assistants, and conversational consumer applications.
These scenarios can benefit significantly from low latency, natural turn-taking, and richer voice interaction.
At the same time, high-volume contact centers need to pay close attention to audio processing costs because the cost per minute can become a major factor at scale.
The most interesting direction is not necessarily a complete migration from cascaded to native.
It is hybrid, orchestrated architecture.
The idea is simple: use native S2S where natural interaction matters most, and use cascaded or text-based components where deterministic reasoning, external data, security, and strict controls are required.
Each architecture does what it does best.
Consider a simple interaction.
The user says:
“Hello, how are you?”
There is no reason to route that interaction through a heavy RAG and enterprise-API pipeline.
The native model can handle it directly, along with greetings, casual conversation, and simple emotional exchanges.
Now consider:
“I want to increase my credit card limit.”
The situation changes completely.
The system may now need authentication, customer data, policy checks, database access, deterministic business rules, and compliance controls.
Instead of allowing the native model to handle the entire workflow, the orchestration layer can route the request to a controlled text-based LLM + RAG pipeline.
Simple interactions stay fast and natural.
Complex business operations move into a more controlled and auditable environment.
This is the real value of smart routing.
Another emerging architectural concept is audio guardrails.
Native models can generate audio directly, which makes traditional text-only moderation less straightforward.
One possible approach is to introduce lightweight security models that analyze the generated audio stream in real time.
These systems could detect potentially inappropriate content, sensitive information, data leakage, or other policy violations and interrupt or block the output before it reaches the user.
The important idea is that security does not disappear when text is removed from the real-time path.
Instead, security controls need to evolve with the architecture.
The evolution of Voice AI isn’t really about old technology being replaced by new technology.
It is a shift in architecture.
Cascaded systems separate speech recognition, reasoning, and speech generation into independent components. That separation provides flexibility, control, security, easier debugging, provider choice, and opportunities for cost optimization.
Native multimodal systems take a different approach. They process audio more directly, reduce the need for multiple transformation stages, and can preserve acoustic information that may otherwise be lost during speech-to-text conversion.
The result can be a much more natural conversational experience.
But native systems bring their own challenges: audio-token consumption, context-window pressure, hardware requirements, more complex guardrails, harder debugging, tool-use orchestration, and enterprise integration.
So the future of Voice AI is unlikely to be decided by choosing a winner between Cascaded and Native Multimodal.
It will be decided by orchestration.
Native models where latency and natural interaction are the point.
Cascaded pipelines where security, determinism, compliance, and enterprise integration are non-negotiable.
The best Voice AI architecture won’t necessarily be the one running the most advanced model.
It will be the one that balances latency, intelligence, security, cost, and control for the specific problem in front of it.
The next generation of Voice AI probably won’t be fully native.
It will be intelligently hybrid.
If you’re building Voice AI in a regulated industry, I’d love to hear how you’re handling this trade-off in the comments — are you leaning cascaded, native, or already routing between both?
2000ms vs. 250ms: The Hidden Architecture War Behind Every Voice AI Product was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.