How Do You Design a Voice AI Architecture That Achieves ~500ms First-Token Latency While Remaining Cost-Effective?

An AI developer is seeking production-grade advice on designing a real-time voice AI architecture that achieves first-token latency of roughly 500 ms or less while remaining cost-effective and scalable. The developer asks which pipeline components — audio transport (WebRTC, WebSocket), voice activity detection, speech-to-text (Deepgram, Gladia, AssemblyAI), LLMs (Gemini Live, OpenAI Realtime, Qwen Omni), text-to-speech (ElevenLabs, Cartesia, Telnyx, OpenAI), and orchestration frameworks (LiveKit, Pipecat) — contribute most to latency, and whether a speech-to-speech model outperforms a traditional STT → LLM → TTS pipeline. The request calls for real-world latency numbers, benchmarks, and lessons learned from production voice agent deployments.

Hi everyone, I’m an AI developer working on a real-time voice AI system and would appreciate advice from people who have experience building low-latency conversational agents. My goal is to achieve: First response token in around 500 ms or less Good speech quality and natural conversations Cost-effective architecture that can scale Support for real-time streaming audio I’m trying to understand the best architecture and component choices across the entire pipeline: Audio transport WebRTC, WebSocket, etc. Voice Activity Detection / End-of-Utterance detection Speech-to-Text Deepgram, Gladia, AssemblyAI, etc. LLMs Gemini Live, OpenAI Realtime, Qwen Omni, custom pipelines, etc. Text-to-Speech ElevenLabs, Cartesia, Telnyx, OpenAI, etc. Orchestration frameworks LiveKit, Pipecat, custom architecture For those who have built production-grade voice agents: What architecture are you using to achieve the lowest possible latency? Which components contribute the most to latency? Is a speech-to-speech model better than a traditional STT → LLM → TTS pipeline? What first-token latency are you seeing in production? Which providers offer the best balance of latency, quality, and cost? Are there any architectural mistakes that commonly increase latency? I’d love to hear real-world numbers, benchmarks, and lessons learned from production deployments. Thanks