Voice agents do not usually fail because the model is “not smart enough.” They fail in the awkward half-second where the user s, breathes, corrects themselves, or interrupts while the AI is still talking.
That tiny moment decides whether the product feels useful or robotic.
If your live AI call cuts people off, talks over them, ignores barge-in, or waits so long that users repeat themselves, no prompt will save the experience. The fix is not one magic model. It is a turn-taking system: audio signals, semantic checks, interruption rules, streaming, and metrics that work together.
This guide walks through a practical voice agent turn-taking design you can ship in a real product.
Text chat is forgiving. A user types. The model answers. If the response takes two seconds, the user may still wait.
Voice is different. Humans expect conversation to move quickly. A delay feels like confusion. An early response feels rude. Talking over the user feels broken.
A production voice agent has to answer three questions again and again:
Most teams start with a simple pipeline:
Microphone -> Speech-to-text -> LLM -> Text-to-speech -> Speaker
That is enough for a demo. It is not enough for a live workflow where the caller changes their mind, uses filler words, speaks in a noisy room, or interrupts because the AI misunderstood them.
The practical goal is not “lowest latency at any cost.” The goal is comfortable turn timing: fast enough to feel alive, patient enough to avoid cutting people off, and interruptible enough to recover when the user takes control.
Recent AI platform activity points toward live agents moving from demos into production workflows:
That creates a useful content gap: builders do not only need “make it faster.” They need a concrete way to decide when the AI should speak, wait, , resume, or hand off.
Think of turn-taking as a small control plane beside your voice pipeline.
Audio stream
-> Voice activity detection
-> Partial transcript stream
-> End-of-turn detector
-> Interruption policy
-> Agent state machine
-> Response planner
-> Streaming TTS
Each layer answers a different question.
| Layer | Job | Common failure |
|---|---|---|
| VAD | Detect speech vs silence | Background noise triggers false speech |
| Endpointing | Decide when a turn may be over | Cuts off slow speakers |
| Semantic end-of-turn | Check whether the thought is complete | Waits too long on short answers |
| Barge-in | Let user interrupt AI speech | User speaks but AI keeps talking |
| State machine | Track listen/respond/ states | Race conditions between audio and tools |
| Metrics | Measure timing and recovery | Team optimizes averages while p95 is bad |
You can buy pieces of this stack from providers, but you still need product-specific policy. A banking workflow, medical triage flow, coding assistant, and onboarding agent should not share the same interruption behavior.
Start with explicit states. Do not let every async callback mutate the session freely.
LISTENING
user speech starts -> USER_SPEAKING
USER_SPEAKING
possible end detected -> THINKING
noise rejected -> LISTENING
THINKING
response ready -> AGENT_SPEAKING
user resumes -> USER_SPEAKING
AGENT_SPEAKING
user barge-in -> INTERRUPTED
speech complete -> LISTENING
INTERRUPTED
stop TTS -> USER_SPEAKING
A basic TypeScript sketch:
type CallState =
| 'LISTENING'
| 'USER_SPEAKING'
| 'THINKING'
| 'AGENT_SPEAKING'
| 'INTERRUPTED';
type Event =
| { type: 'speech_started'; at: number; confidence: number }
| { type: 'speech_ended'; at: number; transcript: string }
| { type: 'partial_transcript'; text: string; at: number }
| { type: 'barge_in'; at: number; confidence: number }
| { type: 'response_ready'; responseId: string }
| { type: 'tts_done'; responseId: string };
function transition(state: CallState, event: Event): CallState {
if (state === 'LISTENING' && event.type === 'speech_started') {
return event.confidence > 0.65 ? 'USER_SPEAKING' : 'LISTENING';
}
if (state === 'USER_SPEAKING' && event.type === 'speech_ended') {
return 'THINKING';
}
if (state === 'THINKING' && event.type === 'speech_started') {
return 'USER_SPEAKING';
}
if (state === 'THINKING' && event.type === 'response_ready') {
return 'AGENT_SPEAKING';
}
if (state === 'AGENT_SPEAKING' && event.type === 'barge_in') {
return event.confidence > 0.75 ? 'INTERRUPTED' : 'AGENT_SPEAKING';
}
if (state === 'AGENT_SPEAKING' && event.type === 'tts_done') {
return 'LISTENING';
}
if (state === 'INTERRUPTED') {
return 'USER_SPEAKING';
}
return state;
}
The exact states can change, but the discipline matters. Voice systems are streaming systems. Without a state machine, you will eventually play stale audio after the user has already corrected the agent.
Silence alone is a weak signal.
A user might because they are thinking. They might say “I need to book a flight from Delhi to…” and before giving the city. If your agent jumps in, the call feels clumsy.
A better end-of-turn detector combines:
Example policy:
type TurnSignal = {
silenceMs: number;
transcript: string;
intentConfidence: number;
requiredSlotsFilled: boolean;
lastAgentQuestionType: 'yes_no' | 'slot_fill' | 'open';
};
function shouldEndTurn(signal: TurnSignal): boolean {
const text = signal.transcript.trim().toLowerCase();
if (signal.lastAgentQuestionType === 'yes_no') {
return signal.silenceMs > 250 && /^(yes|no|yeah|nope|correct|right)\b/.test(text);
}
if (signal.lastAgentQuestionType === 'slot_fill') {
return signal.silenceMs > 450 && signal.requiredSlotsFilled;
}
const looksIncomplete = /\b(and|or|from|to|because|with|for)$/i.test(text);
if (looksIncomplete) return false;
return signal.silenceMs > 700 && signal.intentConfidence > 0.7;
}
This is intentionally simple. You can replace the regex with a classifier later. The key point is that end-of-turn is not only an audio problem. It is a conversation problem.
Barge-in means the user can interrupt while the AI is speaking.
Many teams treat it as a boolean setting: on or off. That is too crude.
A production system should decide what kind of interruption happened:
These should not all behave the same.
type BargeInDecision = 'ignore' | 'duck_audio' | '_and_listen' | 'stop_and_reset';
function classifyBargeIn(text: string, confidence: number): BargeInDecision {
const clean = text.trim().toLowerCase();
if (confidence < 0.6) return 'ignore';
if (/^(mm|mhm|uh huh|yeah|okay)$/.test(clean)) return 'duck_audio';
if (/\b(stop|cancel|never mind|hold on)\b/.test(clean)) return 'stop_and_reset';
if (/\b(no|actually|wait|i mean|that's wrong)\b/.test(clean)) return '_and_listen';
return '_and_listen';
}
For long responses, consider audio ducking before a full stop. Ducking lowers the AI voice while the system decides whether the user is truly taking the turn. This avoids abrupt cutoffs when the user only says “yeah.”
One common bug: the LLM generates a long answer, TTS starts streaming it, then the user interrupts, but the old response keeps leaking into the call.
Avoid this by separating the response plan from the audio stream.
Each response should have an ID, a cancel token, and a current validity check.
class SpeechController {
private activeResponseId: string | null = null;
start(responseId: string) {
this.activeResponseId = responseId;
}
shouldPlay(responseId: string) {
return this.activeResponseId === responseId;
}
cancel(responseId: string) {
if (this.activeResponseId === responseId) {
this.activeResponseId = null;
}
}
}
Before each TTS chunk plays, check whether the response is still active. If the user interrupted, drop queued chunks immediately.
This also helps when tool calls finish late. A booking lookup from the previous user intent should not speak after the user has already changed the destination.
You cannot tune turn-taking with one total latency number. Break it down.
A practical first budget for a responsive voice workflow:
| Stage | Target | Notes |
|---|---|---|
| VAD speech-start detection | 50-120 ms | Fast enough for barge-in |
| End-of-turn decision | 250-800 ms | Depends on question type |
| First LLM token or plan | 150-500 ms | Use smaller models for routing when possible |
| First TTS audio chunk | 100-300 ms | Stream early, do not wait for full answer |
| Tool call acknowledgement | < 500 ms | Say what is happening if the tool is slow |
The trick is overlap. While the user is speaking, stream partial transcripts. While the end-of-turn detector waits, prepare likely intents. While the LLM streams, send short TTS chunks.
But be careful: overlapping work creates stale work. Every async stage needs cancellation.
Voice agents often call tools: search a knowledge base, update a CRM, schedule a meeting, refund an invoice, or open a support ticket.
Turn-taking becomes riskier when speech triggers actions.
Use three rules:
type ToolRequest = {
intentVersion: number;
toolName: string;
args: Record<string, unknown>;
reversible: boolean;
};
function canExecuteTool(currentIntentVersion: number, req: ToolRequest) {
if (req.intentVersion !== currentIntentVersion) return false;
if (!req.reversible) return false; // require confirmation elsewhere
return true;
}
This is especially important for builders shipping AI workflows into customer-facing products. Users speak casually. They revise themselves. Your system has to treat speech as evolving input until the turn is stable.
Average latency is not enough.
Track these metrics per conversation, per environment, and per user segment:
A simple event log helps:
{
"call_id": "call_123",
"turn_id": "turn_009",
"events": [
{ "type": "speech_started", "t": 1200 },
{ "type": "speech_ended", "t": 4100 },
{ "type": "agent_audio_started", "t": 4620 },
{ "type": "barge_in", "t": 5300 },
{ "type": "agent_audio_stopped", "t": 5410 }
],
"metrics": {
"end_to_audio_ms": 520,
"barge_in_stop_ms": 110
}
}
Review bad calls weekly. The fastest way to improve turn-taking is to listen to the exact moments where the user had to repeat, correct, or wait.
Not every turn deserves the same silence threshold.
Use different settings for different moments:
| Conversation moment | Better behavior |
|---|---|
| Yes/no question | Respond quickly after short answer |
| Address, email, or ID capture | Wait longer; users speak in chunks |
| Emotional complaint | Leave more space; avoid rushing |
| Confirmation before action | Require complete answer and explicit consent |
| Long explanation by agent | Enable barge-in aggressively |
| Background-noise environment | Raise speech confidence threshold |
A voice agent that handles an angry support call should not interrupt like a fast command palette. Context matters.
A faster agent that interrupts users is worse than a slightly slower agent that listens well. Optimize for completed turns, not benchmark bragging rights.
A 300 ms may be enough after “yes.” It is not enough after “my account number is…” Use adaptive thresholds.
When the user interrupts, old audio must stop. Cancel queued chunks, tool summaries, and delayed follow-ups tied to the previous intent.
People say “yeah,” “right,” and “mm-hmm” while listening. Do not reset the whole conversation every time.
Use this before you ship a live voice agent:
Voice agent turn-taking is the system that decides when the user is speaking, when the user is done, when the AI should respond, and how the AI should behave if the user interrupts.
Barge-in lets a user interrupt an AI voice agent while it is speaking. A good implementation stops or lowers the AI audio, listens to the user, and updates the conversation state without losing context.
No. Latency is about speed. Turn-taking is about timing and control. A low-latency agent can still feel bad if it cuts users off or ignores interruptions.
It depends on the conversation. A yes/no answer may need only a short . A form field, address, or emotional explanation needs more patience. Use adaptive thresholds instead of one global silence value.
Usually yes for long spoken responses, but interruption should be classified. Backchannels like “mm-hmm” may only require audio ducking. Corrections and cancellation should or stop the response.
Test with noisy audio, slow speakers, interruptions, corrections, accents, long tool calls, and users who change their mind mid-sentence. Measure false cutoffs, ignored interruptions, repeat rate, and barge-in stop time.