{"slug": "voice-ai-latency-why-faster-isn-t-always-better", "title": "Voice AI Latency: Why Faster Isn't Always Better", "summary": "A developer found that optimizing voice AI latency to a p95 of 880ms caused the system to interrupt users mid-sentence, because the AI was too fast to respond to brief thinking pauses. The fix involved disabling eager end-of-turn detection and increasing voice endpoint thresholds, which added about half a second of delay but made conversations feel natural. The key insight is that processing latency and listening patience must be balanced to avoid frustrating users.", "body_md": "# Voice AI Latency: Why Faster Isn't Always Better\n\nMy pipeline was hitting a p95 of about 880ms from the end of a turn to the first audio byte. On paper, that's great. In practice, it was a disaster. The AI was constantly interrupting users mid-sentence because it was *too* fast to respond to a brief thinking pause.\n\n## The Latency-Endpointing Trap\n\nThe core issue is that latency and endpointing are linked. When your pipeline is incredibly fast, any mistake the Voice Activity Detection (VAD) makes in deciding the turn has ended is amplified. If the AI responds in under a second, it cuts the user off. If it were slightly slower, the user would have resumed speaking, the \"barge-in\" mechanism would have triggered, and the conversation would have felt natural.\n\nI was treating endpointing as a minor config detail, but it's actually the primary driver of conversational feel.\n\n## Practical Tuning for Natural Conversation\n\nTo fix the interruptions, I moved away from aggressive settings. Here is the a/b test result of the config changes I implemented:\n\n**Eager end-of-turn:** Switched from ON (0.35) to**OFF****VOICE_EOT_THRESHOLD:** Increased from 0.5 to**0.7****VOICE_EOT_TIMEOUT_MS:** Increased from 700ms to**1200ms**\n\nThe result? The \"cut-ins\" stopped. The response now arrives about half a second later, which feels like a natural human beat rather than a technical lag.\n\n## Analyzing the p95 Tail\n\nWhen auditing your AI workflow, stop looking at p50 (median) latency. A \"sub-second\" p50 is a vanity metric; the p95 is what users actually experience. This is where conversations break down. My current breakdown looks like this:\n\n**Deepgram Flux STT:**~350 ms** VAD fallback:**~280 ms** Cartesia Sonic TTS:**~250 ms** Total p95:**~880 ms\n\nTo keep the experience snappy without feeling robotic, I use a specific optimization: chunking the first TTS request. Instead of buffering a full sentence, I send the first speakable fragment to Cartesia as soon as I have 4 to 8 tokens (capped at a 250ms wait). This shaves off perceived latency without triggering the \"interruption\" bug.\n\nThe biggest takeaway for anyone building a voice agent from scratch: distinguish between \"processing latency\" (the time to generate a response) and \"listening patience\" (the time the AI waits before speaking). Optimizing the former while ignoring the latter is a recipe for a frustrated user base.\n\n[Next Streamlit Sales Dashboard: A Practical Tutorial →](/en/threads/3098/)", "url": "https://wpnews.pro/news/voice-ai-latency-why-faster-isn-t-always-better", "canonical_source": "https://promptcube3.com/en/threads/3112/", "published_at": "2026-07-25 08:00:51+00:00", "updated_at": "2026-07-25 08:08:46.195179+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-products", "ai-tools", "natural-language-processing"], "entities": ["Deepgram Flux STT", "Cartesia Sonic TTS", "Voice Activity Detection"], "alternates": {"html": "https://wpnews.pro/news/voice-ai-latency-why-faster-isn-t-always-better", "markdown": "https://wpnews.pro/news/voice-ai-latency-why-faster-isn-t-always-better.md", "text": "https://wpnews.pro/news/voice-ai-latency-why-faster-isn-t-always-better.txt", "jsonld": "https://wpnews.pro/news/voice-ai-latency-why-faster-isn-t-always-better.jsonld"}}