How We Got Voice AI Response Times Under 1.8 Seconds on Real Phone Calls

Minions.AI, a voice-based service dispatch platform for trade contractors, reduced voice AI response times on real phone calls from ~2,900ms to an average of 1,450ms (sub-1.8s in 99% of live calls). Developer Parvej Shah replaced the sequential pipeline with overlapping event-driven streams, using neural voice activity detection on 20ms frames, streaming token-based TTS, and edge WebSocket termination to cut latency.

By Parvej Shah — Full-Stack Web Developer & AI Systems Engineer based in Dhaka, Bangladesh. There's a specific type of frustration that's hard to explain unless you've experienced it. You call a business. An automated voice picks up. You ask your question. And then — silence. Not a brief pause. A real silence. Long enough that you start wondering if the call dropped, long enough that you pull the phone away from your ear to check the signal bars. That silence is what we were trying to eliminate when building the telephony dispatcher for Minions.AI , a voice-based service dispatch platform for trade contractors. In human conversation, the natural gap between one person finishing a sentence and the other beginning a response is around 200 to 300 milliseconds. Anything beyond 600ms starts to feel awkward. At 2,500ms — which was where the original prototype sat — callers would repeat themselves, raise their voice, or hang up. The call experience was technically functional and practically unusable. The first design was a completely natural one: record audio, run transcription, generate a response, synthesize speech, play it back. Each stage waited for the previous one to finish. The latency budget looked like this: | Stage | Time | |---|---| | Voice Activity Detection end-of-turn | 800ms | | Speech-to-Text transcription | 400ms | | LLM generation full response | 1,200ms | | Text-to-Speech synthesis | 500ms | Total | ~2,900ms | That math is catastrophic for a phone call. And it gets worse in real conditions: cellular networks introduce jitter, LLM response times have variance, TTS output buffering adds overhead. The solution wasn't to make each stage faster in isolation. It was to stop treating them as stages at all. The rewrite changed the mental model from a sequential pipeline to an overlapping set of event-driven streams. Nothing waits for anything it doesn't strictly have to. The original design waited for 800ms of audio silence before assuming the caller had finished speaking. We replaced this with a WebRTC-compatible neural Voice Activity Detection model running on 20ms audio frames. It detects speech completion at the prosodic level — reading the natural falling intonation of a completed sentence — rather than just measuring decibels. Instead of waiting for the full LLM completion before initiating TTS, we stream tokens into a buffer that fires TTS synthesis the moment a punctuation mark period, comma, question mark is encountered. The user hears the first sentence while the LLM is still reasoning about the second. Audio packets are processed over raw WebSockets terminating at edge nodes closest to the telecom provider, stripping out over 120ms of round-trip latency. The resulting latency dropped from ~2,900ms to an average of 1,450ms sub-1.8s in 99% of live calls . Written by Parvej Shah , Full-Stack Web Developer & AI Systems Engineer based in Dhaka, Bangladesh. Explore more case studies and engineering insights at parvejshah.com.