Hi everyone,
I’m an AI developer working on a real-time voice AI system and would appreciate advice from people who have experience building low-latency conversational agents.
My goal is to achieve:
First response token in around 500 ms or less
Good speech quality and natural conversations
Cost-effective architecture that can scale
Support for real-time streaming audio
I’m trying to understand the best architecture and component choices across the entire pipeline:
Audio transport (WebRTC, WebSocket, etc.)
Voice Activity Detection / End-of-Utterance detection
Speech-to-Text (Deepgram, Gladia, AssemblyAI, etc.) LLMs (Gemini Live, OpenAI Realtime, Qwen Omni, custom pipelines, etc.)
Text-to-Speech (ElevenLabs, Cartesia, Telnyx, OpenAI, etc.) Orchestration frameworks (LiveKit, Pipecat, custom architecture)
For those who have built production-grade voice agents: What architecture are you using to achieve the lowest possible latency?
Which components contribute the most to latency?
Is a speech-to-speech model better than a traditional STT → LLM → TTS pipeline?
What first-token latency are you seeing in production?
Which providers offer the best balance of latency, quality, and cost?
Are there any architectural mistakes that commonly increase latency?
I’d love to hear real-world numbers, benchmarks, and lessons learned from production deployments.
Thanks!