I made my voice agent slower on purpose

A developer running a voice AI sales training platform intentionally slowed down their voice agent's pipeline to fix an interruption problem. The pipeline's p95 latency of 880 ms was fast enough that premature endpointing caused the AI to cut off users mid-sentence. By adjusting endpointing settings and disabling eager end-of-turn, the interruptions stopped and the reply now lands as a natural beat. The developer warns that speed amplifies endpointing errors and that p95 latency, not median, is what users actually feel.

My pipeline was fast enough to interrupt people. Here is what I changed, what it cost, and the second bug that made calls go deaf without anyone noticing. I run a voice AI sales training platform. Users practice cold calls against an AI buyer, in real time, over WebRTC. The stack is Pipecat for orchestration, LiveKit for transport, Deepgram Flux for STT, Cartesia Sonic for TTS, and a small fast model over OpenRouter for the conversation. In July I had a pipeline I was proud of: p95 of about 880 ms from confirmed end of turn to the first audio byte coming back. Then I listened to real calls, and the AI buyer kept talking over people. Not at natural turn boundaries. Mid-sentence. A rep would say "so what we do is..." , pause half a second to think, and the buyer would answer the unfinished thought. The pipeline was working exactly as designed. That was the problem. My endpointing was pinned at the aggressive end of the config range: | Setting | Value | |---|---| VOICE EOT THRESHOLD | 0.5 | VOICE EOT TIMEOUT MS | 700 | | Eager end-of-turn | on, at 0.35 | | Turn-commit buffer | 0 ms | A one-second thinking pause was enough to commit the turn. Then an 880 ms pipeline delivered a full reply before the caller got going again. Here is the part I had not internalized. If my pipeline had been slower, the same premature endpointing would have been less noticeable. The reply would have landed after the caller resumed, barge-in would have caught it, and the turn would have recovered. Being fast is what converted a detection error into an interruption. Latency and endpointing are not independent. Every millisecond you shave off the pipeline raises the cost of a premature end-of-turn decision. Speed is not free. It amplifies your endpointing errors. Almost every voice latency post I have read treats the pipeline as the thing to optimize and endpointing as a config detail. In my experience it is the reverse. The pipeline was the easy part. I ran a live A/B on staging and production on July 15: | Setting | Before | After | |---|---|---| | Eager end-of-turn | on 0.35 | off | VOICE EOT THRESHOLD | 0.5 | 0.7 | VOICE EOT TIMEOUT MS | 700 | 1200 | The cut-ins stopped. The reply now lands roughly half a second later than before, and it reads as a natural beat rather than a delay. Nobody has complained that the agent is slow. Plenty of people had complained that it interrupted. Then I rewrote the constraint in my own docs, which is the actual takeaway: Processing p95 must stay under 1 second from the confirmed end of turnto first audio. Listening patience is deliberate and sitsoutsidethat budget. Two different numbers, with two different failure modes. Conflating them is how you end up optimizing the one that was already fine. | Stage | Contribution | |---|---| | Deepgram Flux STT | ~350 ms | | VAD fallback | ~280 ms | | Cartesia Sonic TTS | ~250 ms | Total, p95 | ~880 ms | Some honesty about where that actually sits. Typical production voice deployments in 2026 land around 680 ms p50 and 1,180 ms p95. Well-optimized stacks reach sub-250 ms p50. So 880 ms p95 is solid, not remarkable, and I am quoting the tail rather than the median. I mention it because a lot of latency posts quote p50 and call it "sub-second." If you are comparing stacks, insist on the percentile. The p95 is what your users actually feel, because the tail is exactly where a conversation breaks down. A median that looks great and a p95 that does not is an agent that feels fine in a demo and frustrating in production. One small thing that genuinely helped: chunking the first TTS request. I send the first speakable fragment to Cartesia as soon as I have 4 to 8 tokens, capped at a 250 ms wait, rather than buffering a full sentence. It costs nothing and buys a few hundred milliseconds of perceived responsiveness on every single turn. The Deepgram Flux websocket is long-lived, and it will occasionally drop abruptly mid-call. The signature in my logs was no close frame received or sent . Rare, but it showed up in Sentry as a class of failure, not a one-off. My code had a startup error latch. Its original job was reasonable: if the connect probe fails at session start, fall back to nova-3. But run stt treated that latch as permanent . So a mid-call drop that also failed a single reconnect attempt set the latch, and the agent was deaf for the rest of the call while continuing to hold a conversation from context. Two latent hazards made it considerably worse: The watchdog got pinned off by the failure itself. If the drop happened mid-user-turn, flux user speaking stayed stuck true. My inactivity watchdog skips its checks while either party is speaking. So a deaf call would never time out. It would run forever, burning credits and vendor spend, with a user talking to something that had stopped listening minutes earlier. A dead connection that reported itself healthy. A non-connection error in the receive loop killed transcription while the socket object still said open. is connected returned true. Nothing anywhere detected it. is connected returns false even when the socket object claims otherwise. run stt path retries UserStoppedSpeakingFrame , which unpins the watchdog. start now swallows a transient connect failure with a warning so the StartFrame still propagates. Raising there left every downstream processor unstarted. Retry forever is a deliberate decision. An outage of any length recovers the moment Deepgram is reachable again. An outage that never recovers gets ended by the inactivity watchdog, not by a latch. Latches are how you end up with permanently deaf calls. Logging policy matters as much as the retry logic: a warning on every attempt, exactly one Sentry error per outage fired on the third consecutive failure , and an info line with total outage duration on recovery. That gives you real signal on sustained outages without drowning in transient blips. Zero cost on the healthy path, because all of it only runs when is connected is already false. Eleven new tests, suite at 690 passing. The counterintuitive lesson underneath all three: my users never asked for a faster agent. They asked for one that would let them finish a sentence. I build real-time voice systems: Pipecat, LiveKit, Deepgram, Cartesia. The product above is ConvoSparr. I am currently taking on voice AI contract work, so if you are fighting latency, turn-taking, or reliability in a production voice agent, get in touch: karthik@convosparr.com