Building a voice agent is not simply a matter of connecting speech-to-text to an LLM and adding text-to-speech.
A production voice agent has to solve a harder problem:
How do you make an AI system listen, reason, use tools, remember context, and respond quickly enough that the conversation still feels natural?
LangChain can handle the agent and tool-orchestration layer, but the realtime experience depends heavily on what happens around it.
A practical architecture looks like this:
User microphone
↓
Audio streaming
↓
Speech-to-Text (STT)
↓
Transcript / turn detection
↓
LangChain Agent
↓
Tools / APIs / Business Logic
↓
Streaming response
↓
Text-to-Speech (TTS)
↓
User hears response
LangChain's current voice-agent documentation describes this as the "sandwich" architecture: STT → agent → TTS. The advantage is that each layer can be replaced independently, while the agent can continue using the broader LangChain ecosystem.
Before writing code, separate the voice agent into five responsibilities:
This separation matters because these components have different performance characteristics.
For example, changing your TTS provider should not require rewriting your business logic. Similarly, changing the LLM should not require rebuilding your audio transport.
That modularity is one of the strongest reasons to use a cascaded architecture instead of putting everything into one model.
There are two major ways to build a voice agent.
Audio
↓
STT
↓
Text
↓
LangChain Agent
↓
Text
↓
TTS
↓
Audio
This gives you control over every component.
You can choose one STT provider, another LLM, and a completely different TTS provider.
It also makes debugging easier because you can inspect the transcript, agent decision, tool call, and final response independently.
The trade-off is additional infrastructure and potential latency.
Audio
↓
Multimodal Voice Model
↓
Audio
This can reduce the number of moving pieces and can preserve more information about how something was spoken, such as tone.
However, it can reduce your control over individual components and introduce provider-specific constraints.
For business applications where tool execution, observability, provider flexibility, and deterministic workflows matter, the cascaded architecture remains highly practical.
This is where many voice-agent implementations go wrong.
A naive implementation waits for the entire chain:
Record entire sentence
↓
Transcribe
↓
Wait for complete LLM response
↓
Generate complete audio
↓
Play response
The user experiences one long delay.
A streaming architecture instead looks like:
Audio chunk
↓
STT starts immediately
↓
Transcript arrives
↓
Agent starts generating
↓
First response tokens arrive
↓
TTS starts
↓
Audio starts playing
The system does not wait for every stage to finish before the next stage begins.
LangChain's official voice-agent example uses asynchronous streaming and RunnableGenerator
to connect STT, the agent, and TTS. The documentation notes that this pipeline can achieve sub-700 ms latency with suitable STT and TTS providers.
The important lesson is:
Realtime voice is primarily a pipeline-design problem, not just a model-selection problem.
Research on realtime voice agents similarly identifies streaming and pipelining across STT, LLM, and TTS as a central mechanism for reducing perceived latency.
Once speech has been converted into text, the voice layer can hand the request to a normal LangChain agent.
Current LangChain applications use create_agent
as the primary entry point.
A minimal agent can look like this:
from langchain.agents import create_agent
def check_order_status(order_id: str) -> str:
"""Return the current status of an order."""
return f"Order {order_id} is currently being processed."
agent = create_agent(
model="openai:gpt-5.4",
tools=[check_order_status],
system_prompt="""
You are a customer support voice agent.
Keep spoken responses short.
Ask for missing information instead of guessing.
Use tools whenever the user asks for account-specific information.
"""
)
The important part is not the five lines of code.
It is the tool boundary.
A voice agent should not directly manipulate your database or business systems through arbitrary model-generated text.
Instead:
User:
"Where is order 4821?"
↓
Agent
↓
check_order_status("4821")
↓
Business system
↓
Structured result
↓
Agent
↓
"Your order is currently being processed."
LangChain agents can reason over available tools and execute them as part of the agent loop. The current agent implementation is built on LangGraph's runtime.
This is an overlooked part of voice-agent engineering.
A tool that works well for a text chatbot may be poorly designed for a voice agent.
For example, avoid giving the agent a tool that returns:
{"customer_id": 1827,
"subscription_status": "active",
"plan": "enterprise",
"billing_cycle": "annual",
"last_payment": "...",
"payment_method": "..."}
if the only thing the user asked was:
"Is my subscription active?"
Instead, make the tool return information that the agent can quickly reason over.
def get_subscription_status(customer_id: str) -> str:
"""Check whether a customer's subscription is active."""
...
The voice agent can then respond:
"Yes, your subscription is active."
The rule is simple:
Design tools around decisions, not database tables.
This reduces unnecessary reasoning and makes spoken responses easier to control.
A language model optimized for written chat can produce paragraphs.
A voice agent should not.
Compare:
Chatbot response:
"Certainly. I can help you with that. According to the information available in your account, your order has been processed successfully and is currently in transit. You can expect delivery within the next two to three business days..."
Voice response:
"Your order is in transit. It should arrive within two to three business days."
Voice requires a different response policy.
A useful system instruction is:
You are a voice assistant.
Speak naturally and concisely.
Prefer one or two sentences per response.
Do not read JSON, URLs, IDs, tables, or long lists aloud.
Ask one question at a time.
If a tool fails, explain the problem briefly and offer the next action.
Never invent information that is unavailable from a tool.
This is not merely prompt optimization.
It is interface design.
Voice conversations become awkward if the agent forgets what was said five seconds earlier.
Consider:
User: "I want to book an appointment tomorrow."
Agent: "What time?"
User: "Around 4."
Please ensure the agent understands that "4" refers to the appointment.
LangChain's voice-agent example uses conversation state with a checkpointer and a unique thread ID so the agent can retain context across turns.
Conceptually:
User
↓
Voice session ID
↓
Conversation state
↓
LangChain agent
↓
Response
For a production system, distinguish between:
Things said during the current call.
Examples:
Information that should survive the call.
Examples:
Do not put every piece of customer data into the LLM's conversation history.
Retrieve what is needed for the current decision.
This is one of the biggest differences between a chatbot and a voice agent.
Imagine the agent is saying:
"Your appointment is scheduled for Thursday at—"
The user interrupts:
"Actually, make that Friday."
A real voice interface should stop speaking.
That means your system needs to support barge-in.
A simplified flow is:
Agent speaking
↓
User starts talking
↓
Detect interruption
↓
Stop TTS playback
↓
Cancel/ignore remaining audio
↓
Process new user input
Without interruption handling, the system feels less like a conversation and more like an IVR reading a script.
This is why audio transport, turn detection, and cancellation logic are just as important as the LLM.
For a browser-based implementation, WebSockets are a practical transport layer.
The client captures microphone audio:
Browser microphone
↓
PCM audio chunks
↓
WebSocket
↓
Backend
The backend sends synthesized audio back through the same connection:
Backend
↓
TTS audio chunks
↓
WebSocket
↓
Browser
↓
Speaker
LangChain's reference voice application uses WebSockets for bidirectional audio streaming and notes that the same general architecture can be adapted to telephony or WebRTC.
The important design decision is to keep the transport layer independent from the agent.
Your agent should not care whether the request came from:
It should receive an input event and return agent events.
A simplified LangChain pipeline can conceptually look like:
from langchain_core.runnables import RunnableGenerator
pipeline = (
RunnableGenerator(stt_stream)
| RunnableGenerator(agent_stream)
| RunnableGenerator(tts_stream)
)
Each stage consumes and produces a stream.
STT events
↓
Agent events
↓
TTS events
This is more useful than treating the voice agent as one giant function.
Each stage can be measured independently.
For example:
Audio received
↓
STT first transcript 180 ms
↓
Agent first token 220 ms
↓
TTS first audio 160 ms
↓
User hears response ~560 ms
These measurements tell you where the actual bottleneck is.
Do not measure only total API response time.
For voice systems, track at least:
How quickly does the system understand the user's speech?
How quickly does the agent begin responding?
How quickly does the user hear the response?
How long does the agent take to finish speaking?
How long do external API calls take?
For example:
User finishes speaking
│
├── STT: 210 ms
│
├── Agent starts: 35 ms
│
├── CRM API: 420 ms
│
├── LLM first token: 180 ms
│
└── TTS first audio: 140 ms
If your first-audio latency is 1.2 seconds, changing the LLM may not solve the problem if the real bottleneck is a 700 ms CRM API.
Voice agents expose slow backend systems immediately.
Imagine:
Voice input
↓
Agent
↓
CRM
↓
Database
↓
Payment API
↓
Agent
↓
TTS
Even if your LLM is extremely fast, the conversation can feel slow because of downstream services.
Use:
For example, if the agent needs customer information and appointment availability, those lookups may not always need to happen sequentially.
But be careful with parallel execution when tools have side effects.
Reading two systems in parallel is very different from creating two appointments simultaneously.
Voice agents fail differently from chatbots.
Potential failures include:
The agent should have explicit fallback behavior.
For example:
Tool timeout
↓
Retry if operation is safe
↓
Still failing?
↓
Tell the user
↓
Offer alternative action
Never let the model hide a failed transaction by pretending it succeeded.
For actions such as payments, bookings, cancellations, or account changes, the system should verify the actual backend result before confirming completion.
LangChain is useful for:
But LangChain is not your complete voice infrastructure.
You still need to solve:
Think of LangChain as the reasoning and orchestration layer, not the entire voice stack.
A simple voice assistant might only need:
User → Agent → Tool → Response
A business workflow can become more complicated:
Incoming call
↓
Identify customer
↓
Understand intent
↓
Check account
↓
Determine eligibility
↓
Call external system
↓
Human approval?
↙ ↘
Yes No
↓ ↓
Human Complete
review
This is where graph-based orchestration becomes valuable.
LangChain's current create_agent
implementation already uses LangGraph underneath, while direct LangGraph workflows are useful when you need more explicit control over state, branching, persistence, interrupts, or complex workflows.
The important point is:
Do not add LangGraph simply because you are building a voice agent. Add graph-level orchestration when the workflow actually needs it.
A practical production architecture could look like this:
┌──────────────────┐
│ Web / Mobile │
│ / Phone Client │
└────────┬─────────┘
│
Audio Stream
│
▼
┌──────────────────┐
│ Audio Gateway │
│ WebSocket/WebRTC │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ STT │
└────────┬─────────┘
│
Transcript
│
▼
┌──────────────────┐
│ LangChain Agent │
│ │
│ State + Tools │
└───────┬──────────┘
│
┌───────────┼───────────┐
▼ ▼ ▼
CRM/API Database Calendar
│ │ │
└───────────┼───────────┘
│
▼
Agent Response
│
▼
┌──────────────────┐
│ TTS │
└────────┬─────────┘
│
▼
Audio Stream
│
▼
User
This architecture has an important property:
Every layer can evolve independently.
You can replace the STT provider without rebuilding the agent.
You can replace the LLM without rebuilding the audio gateway.
You can replace the CRM without changing the voice interface.
That is what makes the architecture suitable for production.
Building a voice agent with LangChain is not primarily about writing an LLM prompt.
The difficult engineering work is around the LLM:
LangChain gives you a strong agent and tool-orchestration layer, while the voice infrastructure handles the real-time UI interface. Its current documentation demonstrates this separation through a streaming STT → LangChain agent → TTS architecture.
If you’re planning to take this architecture beyond a prototype and build a production-ready voice system with custom workflows, backend integrations, multilingual support, monitoring, and low-latency interactions, explore Ciphernutz’s AI Voice Agent Development services.
The goal is not to make an LLM speak. The goal is to make a business workflow conversational without making it unreliable.