Voice Agent Turn-Taking: Stop Live AI Calls From Talking Over Users A developer outlines a practical turn-taking system for voice agents that prevents awkward interruptions and delays. The design combines voice activity detection, semantic end-of-turn checks, barge-in handling, and a state machine to ensure comfortable conversation timing in production workflows. Voice agents do not usually fail because the model is “not smart enough.” They fail in the awkward half-second where the user pauses, breathes, corrects themselves, or interrupts while the AI is still talking. That tiny moment decides whether the product feels useful or robotic. If your live AI call cuts people off, talks over them, ignores barge-in, or waits so long that users repeat themselves, no prompt will save the experience. The fix is not one magic model. It is a turn-taking system: audio signals, semantic checks, interruption rules, streaming, and metrics that work together. This guide walks through a practical voice agent turn-taking design you can ship in a real product. Text chat is forgiving. A user types. The model answers. If the response takes two seconds, the user may still wait. Voice is different. Humans expect conversation to move quickly. A delay feels like confusion. An early response feels rude. Talking over the user feels broken. A production voice agent has to answer three questions again and again: Most teams start with a simple pipeline: php Microphone - Speech-to-text - LLM - Text-to-speech - Speaker That is enough for a demo. It is not enough for a live workflow where the caller changes their mind, uses filler words, speaks in a noisy room, or interrupts because the AI misunderstood them. The practical goal is not “lowest latency at any cost.” The goal is comfortable turn timing : fast enough to feel alive, patient enough to avoid cutting people off, and interruptible enough to recover when the user takes control. Recent AI platform activity points toward live agents moving from demos into production workflows: That creates a useful content gap: builders do not only need “make it faster.” They need a concrete way to decide when the AI should speak, wait, pause, resume, or hand off. Think of turn-taking as a small control plane beside your voice pipeline. php Audio stream - Voice activity detection - Partial transcript stream - End-of-turn detector - Interruption policy - Agent state machine - Response planner - Streaming TTS Each layer answers a different question. | Layer | Job | Common failure | |---|---|---| | VAD | Detect speech vs silence | Background noise triggers false speech | | Endpointing | Decide when a turn may be over | Cuts off slow speakers | | Semantic end-of-turn | Check whether the thought is complete | Waits too long on short answers | | Barge-in | Let user interrupt AI speech | User speaks but AI keeps talking | | State machine | Track listen/respond/pause states | Race conditions between audio and tools | | Metrics | Measure timing and recovery | Team optimizes averages while p95 is bad | You can buy pieces of this stack from providers, but you still need product-specific policy. A banking workflow, medical triage flow, coding assistant, and onboarding agent should not share the same interruption behavior. Start with explicit states. Do not let every async callback mutate the session freely. php LISTENING user speech starts - USER SPEAKING USER SPEAKING possible end detected - THINKING noise rejected - LISTENING THINKING response ready - AGENT SPEAKING user resumes - USER SPEAKING AGENT SPEAKING user barge-in - INTERRUPTED speech complete - LISTENING INTERRUPTED stop TTS - USER SPEAKING A basic TypeScript sketch: type CallState = | 'LISTENING' | 'USER SPEAKING' | 'THINKING' | 'AGENT SPEAKING' | 'INTERRUPTED'; type Event = | { type: 'speech started'; at: number; confidence: number } | { type: 'speech ended'; at: number; transcript: string } | { type: 'partial transcript'; text: string; at: number } | { type: 'barge in'; at: number; confidence: number } | { type: 'response ready'; responseId: string } | { type: 'tts done'; responseId: string }; function transition state: CallState, event: Event : CallState { if state === 'LISTENING' && event.type === 'speech started' { return event.confidence 0.65 ? 'USER SPEAKING' : 'LISTENING'; } if state === 'USER SPEAKING' && event.type === 'speech ended' { return 'THINKING'; } if state === 'THINKING' && event.type === 'speech started' { return 'USER SPEAKING'; } if state === 'THINKING' && event.type === 'response ready' { return 'AGENT SPEAKING'; } if state === 'AGENT SPEAKING' && event.type === 'barge in' { return event.confidence 0.75 ? 'INTERRUPTED' : 'AGENT SPEAKING'; } if state === 'AGENT SPEAKING' && event.type === 'tts done' { return 'LISTENING'; } if state === 'INTERRUPTED' { return 'USER SPEAKING'; } return state; } The exact states can change, but the discipline matters. Voice systems are streaming systems. Without a state machine, you will eventually play stale audio after the user has already corrected the agent. Silence alone is a weak signal. A user might pause because they are thinking. They might say “I need to book a flight from Delhi to…” and pause before giving the city. If your agent jumps in, the call feels clumsy. A better end-of-turn detector combines: Example policy: type TurnSignal = { silenceMs: number; transcript: string; intentConfidence: number; requiredSlotsFilled: boolean; lastAgentQuestionType: 'yes no' | 'slot fill' | 'open'; }; function shouldEndTurn signal: TurnSignal : boolean { const text = signal.transcript.trim .toLowerCase ; if signal.lastAgentQuestionType === 'yes no' { return signal.silenceMs 250 && /^ yes|no|yeah|nope|correct|right \b/.test text ; } if signal.lastAgentQuestionType === 'slot fill' { return signal.silenceMs 450 && signal.requiredSlotsFilled; } const looksIncomplete = /\b and|or|from|to|because|with|for $/i.test text ; if looksIncomplete return false; return signal.silenceMs 700 && signal.intentConfidence 0.7; } This is intentionally simple. You can replace the regex with a classifier later. The key point is that end-of-turn is not only an audio problem. It is a conversation problem. Barge-in means the user can interrupt while the AI is speaking. Many teams treat it as a boolean setting: on or off. That is too crude. A production system should decide what kind of interruption happened: These should not all behave the same. type BargeInDecision = 'ignore' | 'duck audio' | 'pause and listen' | 'stop and reset'; function classifyBargeIn text: string, confidence: number : BargeInDecision { const clean = text.trim .toLowerCase ; if confidence < 0.6 return 'ignore'; if /^ mm|mhm|uh huh|yeah|okay $/.test clean return 'duck audio'; if /\b stop|cancel|never mind|hold on \b/.test clean return 'stop and reset'; if /\b no|actually|wait|i mean|that's wrong \b/.test clean return 'pause and listen'; return 'pause and listen'; } For long responses, consider audio ducking before a full stop. Ducking lowers the AI voice while the system decides whether the user is truly taking the turn. This avoids abrupt cutoffs when the user only says “yeah.” One common bug: the LLM generates a long answer, TTS starts streaming it, then the user interrupts, but the old response keeps leaking into the call. Avoid this by separating the response plan from the audio stream. Each response should have an ID, a cancel token, and a current validity check. class SpeechController { private activeResponseId: string | null = null; start responseId: string { this.activeResponseId = responseId; } shouldPlay responseId: string { return this.activeResponseId === responseId; } cancel responseId: string { if this.activeResponseId === responseId { this.activeResponseId = null; } } } Before each TTS chunk plays, check whether the response is still active. If the user interrupted, drop queued chunks immediately. This also helps when tool calls finish late. A booking lookup from the previous user intent should not speak after the user has already changed the destination. You cannot tune turn-taking with one total latency number. Break it down. A practical first budget for a responsive voice workflow: | Stage | Target | Notes | |---|---|---| | VAD speech-start detection | 50-120 ms | Fast enough for barge-in | | End-of-turn decision | 250-800 ms | Depends on question type | | First LLM token or plan | 150-500 ms | Use smaller models for routing when possible | | First TTS audio chunk | 100-300 ms | Stream early, do not wait for full answer | | Tool call acknowledgement | < 500 ms | Say what is happening if the tool is slow | The trick is overlap. While the user is speaking, stream partial transcripts. While the end-of-turn detector waits, prepare likely intents. While the LLM streams, send short TTS chunks. But be careful: overlapping work creates stale work. Every async stage needs cancellation. Voice agents often call tools: search a knowledge base, update a CRM, schedule a meeting, refund an invoice, or open a support ticket. Turn-taking becomes riskier when speech triggers actions. Use three rules: type ToolRequest = { intentVersion: number; toolName: string; args: Record