{"slug": "i-made-my-voice-agent-slower-on-purpose", "title": "I made my voice agent slower on purpose", "summary": "A developer running a voice AI sales training platform intentionally slowed down their voice agent's pipeline to fix an interruption problem. The pipeline's p95 latency of 880 ms was fast enough that premature endpointing caused the AI to cut off users mid-sentence. By adjusting endpointing settings and disabling eager end-of-turn, the interruptions stopped and the reply now lands as a natural beat. The developer warns that speed amplifies endpointing errors and that p95 latency, not median, is what users actually feel.", "body_md": "*My pipeline was fast enough to interrupt people. Here is what I changed, what it cost, and the second bug that made calls go deaf without anyone noticing.*\n\nI run a voice AI sales training platform. Users practice cold calls against an AI buyer, in real time, over WebRTC. The stack is Pipecat for orchestration, LiveKit for transport, Deepgram Flux for STT, Cartesia Sonic for TTS, and a small fast model over OpenRouter for the conversation.\n\nIn July I had a pipeline I was proud of: **p95 of about 880 ms** from confirmed end of turn to the first audio byte coming back. Then I listened to real calls, and the AI buyer kept talking over people.\n\nNot at natural turn boundaries. Mid-sentence. A rep would say *\"so what we do is...\"*, pause half a second to think, and the buyer would answer the unfinished thought.\n\nThe pipeline was working exactly as designed. That was the problem.\n\nMy endpointing was pinned at the aggressive end of the config range:\n\n| Setting | Value |\n|---|---|\n`VOICE_EOT_THRESHOLD` |\n0.5 |\n`VOICE_EOT_TIMEOUT_MS` |\n700 |\n| Eager end-of-turn | on, at 0.35 |\n| Turn-commit buffer | 0 ms |\n\nA one-second thinking pause was enough to commit the turn. Then an 880 ms pipeline delivered a full reply before the caller got going again.\n\nHere is the part I had not internalized. **If my pipeline had been slower, the same premature endpointing would have been less noticeable.** The reply would have landed after the caller resumed, barge-in would have caught it, and the turn would have recovered. Being fast is what converted a detection error into an interruption.\n\nLatency and endpointing are not independent. Every millisecond you shave off the pipeline raises the cost of a premature end-of-turn decision. Speed is not free. It amplifies your endpointing errors.\n\nAlmost every voice latency post I have read treats the pipeline as the thing to optimize and endpointing as a config detail. In my experience it is the reverse. The pipeline was the easy part.\n\nI ran a live A/B on staging and production on July 15:\n\n| Setting | Before | After |\n|---|---|---|\n| Eager end-of-turn | on (0.35) | off |\n`VOICE_EOT_THRESHOLD` |\n0.5 | 0.7 |\n`VOICE_EOT_TIMEOUT_MS` |\n700 | 1200 |\n\nThe cut-ins stopped. The reply now lands roughly half a second later than before, and it reads as a natural beat rather than a delay. Nobody has complained that the agent is slow. Plenty of people had complained that it interrupted.\n\nThen I rewrote the constraint in my own docs, which is the actual takeaway:\n\nProcessing p95 must stay under 1 second\n\nfrom the confirmed end of turnto first audio. Listening patience is deliberate and sitsoutsidethat budget.\n\nTwo different numbers, with two different failure modes. Conflating them is how you end up optimizing the one that was already fine.\n\n| Stage | Contribution |\n|---|---|\n| Deepgram Flux STT | ~350 ms |\n| VAD fallback | ~280 ms |\n| Cartesia Sonic TTS | ~250 ms |\nTotal, p95 |\n~880 ms |\n\nSome honesty about where that actually sits. Typical production voice deployments in 2026 land around 680 ms p50 and 1,180 ms p95. Well-optimized stacks reach sub-250 ms p50. So 880 ms p95 is solid, not remarkable, and I am quoting the tail rather than the median.\n\nI mention it because a lot of latency posts quote p50 and call it \"sub-second.\" If you are comparing stacks, insist on the percentile. **The p95 is what your users actually feel,** because the tail is exactly where a conversation breaks down. A median that looks great and a p95 that does not is an agent that feels fine in a demo and frustrating in production.\n\nOne small thing that genuinely helped: chunking the first TTS request. I send the first speakable fragment to Cartesia as soon as I have 4 to 8 tokens, capped at a 250 ms wait, rather than buffering a full sentence. It costs nothing and buys a few hundred milliseconds of perceived responsiveness on every single turn.\n\nThe Deepgram Flux websocket is long-lived, and it will occasionally drop abruptly mid-call. The signature in my logs was `no close frame received or sent`\n\n. Rare, but it showed up in Sentry as a class of failure, not a one-off.\n\nMy code had a `_startup_error`\n\nlatch. Its original job was reasonable: if the connect probe fails at session start, fall back to nova-3. But `run_stt`\n\ntreated that latch as **permanent**. So a mid-call drop that also failed a single reconnect attempt set the latch, and the agent was deaf for the rest of the call while continuing to hold a conversation from context.\n\nTwo latent hazards made it considerably worse:\n\n**The watchdog got pinned off by the failure itself.** If the drop happened mid-user-turn, `_flux_user_speaking`\n\nstayed stuck true. My inactivity watchdog skips its checks while either party is speaking. So a deaf call would never time out. It would run forever, burning credits and vendor spend, with a user talking to something that had stopped listening minutes earlier.\n\n**A dead connection that reported itself healthy.** A non-connection error in the receive loop killed transcription while the socket object still said open. `_is_connected()`\n\nreturned true. Nothing anywhere detected it.\n\n`_is_connected()`\n\nreturns false even when the socket object claims otherwise.`run_stt`\n\npath retries `UserStoppedSpeakingFrame`\n\n, which unpins the watchdog.`start()`\n\nnow swallows a transient connect failure with a warning so the `StartFrame`\n\nstill propagates. Raising there left every downstream processor unstarted.**Retry forever is a deliberate decision.** An outage of any length recovers the moment Deepgram is reachable again. An outage that never recovers gets ended by the inactivity watchdog, not by a latch. Latches are how you end up with permanently deaf calls.\n\nLogging policy matters as much as the retry logic: a warning on every attempt, exactly **one** Sentry error per outage (fired on the third consecutive failure), and an info line with total outage duration on recovery. That gives you real signal on sustained outages without drowning in transient blips.\n\nZero cost on the healthy path, because all of it only runs when `_is_connected()`\n\nis already false. Eleven new tests, suite at 690 passing.\n\nThe counterintuitive lesson underneath all three: my users never asked for a faster agent. They asked for one that would let them finish a sentence.\n\n*I build real-time voice systems: Pipecat, LiveKit, Deepgram, Cartesia. The product above is ConvoSparr. I am currently taking on voice AI contract work, so if you are fighting latency, turn-taking, or reliability in a production voice agent, get in touch: karthik@convosparr.com*", "url": "https://wpnews.pro/news/i-made-my-voice-agent-slower-on-purpose", "canonical_source": "https://dev.to/karthik_r_rk/i-made-my-voice-agent-slower-on-purpose-ci", "published_at": "2026-07-25 04:53:26+00:00", "updated_at": "2026-07-25 05:31:02.582277+00:00", "lang": "en", "topics": ["artificial-intelligence", "developer-tools"], "entities": ["Pipecat", "LiveKit", "Deepgram Flux", "Cartesia Sonic", "OpenRouter"], "alternates": {"html": "https://wpnews.pro/news/i-made-my-voice-agent-slower-on-purpose", "markdown": "https://wpnews.pro/news/i-made-my-voice-agent-slower-on-purpose.md", "text": "https://wpnews.pro/news/i-made-my-voice-agent-slower-on-purpose.txt", "jsonld": "https://wpnews.pro/news/i-made-my-voice-agent-slower-on-purpose.jsonld"}}