Voice AI Latency: Why Faster Isn't Always Better

A developer found that optimizing voice AI latency to a p95 of 880ms caused the system to interrupt users mid-sentence, because the AI was too fast to respond to brief thinking pauses. The fix involved disabling eager end-of-turn detection and increasing voice endpoint thresholds, which added about half a second of delay but made conversations feel natural. The key insight is that processing latency and listening patience must be balanced to avoid frustrating users.

Voice AI Latency: Why Faster Isn't Always Better My pipeline was hitting a p95 of about 880ms from the end of a turn to the first audio byte. On paper, that's great. In practice, it was a disaster. The AI was constantly interrupting users mid-sentence because it was too fast to respond to a brief thinking pause. The Latency-Endpointing Trap The core issue is that latency and endpointing are linked. When your pipeline is incredibly fast, any mistake the Voice Activity Detection VAD makes in deciding the turn has ended is amplified. If the AI responds in under a second, it cuts the user off. If it were slightly slower, the user would have resumed speaking, the "barge-in" mechanism would have triggered, and the conversation would have felt natural. I was treating endpointing as a minor config detail, but it's actually the primary driver of conversational feel. Practical Tuning for Natural Conversation To fix the interruptions, I moved away from aggressive settings. Here is the a/b test result of the config changes I implemented: Eager end-of-turn: Switched from ON 0.35 to OFF VOICE EOT THRESHOLD: Increased from 0.5 to 0.7 VOICE EOT TIMEOUT MS: Increased from 700ms to 1200ms The result? The "cut-ins" stopped. The response now arrives about half a second later, which feels like a natural human beat rather than a technical lag. Analyzing the p95 Tail When auditing your AI workflow, stop looking at p50 median latency. A "sub-second" p50 is a vanity metric; the p95 is what users actually experience. This is where conversations break down. My current breakdown looks like this: Deepgram Flux STT: ~350 ms VAD fallback: ~280 ms Cartesia Sonic TTS: ~250 ms Total p95: ~880 ms To keep the experience snappy without feeling robotic, I use a specific optimization: chunking the first TTS request. Instead of buffering a full sentence, I send the first speakable fragment to Cartesia as soon as I have 4 to 8 tokens capped at a 250ms wait . This shaves off perceived latency without triggering the "interruption" bug. The biggest takeaway for anyone building a voice agent from scratch: distinguish between "processing latency" the time to generate a response and "listening patience" the time the AI waits before speaking . Optimizing the former while ignoring the latter is a recipe for a frustrated user base. Next Streamlit Sales Dashboard: A Practical Tutorial → /en/threads/3098/