Building Real-Time Voice Agents from Scratch

Nemorize has published a learning roadmap titled "Building Real-Time Voice Agents from Scratch," covering the full pipeline from audio fundamentals and speech detection to ASR, LLM streaming, and TTS. The roadmap is divided into five parts, including foundations, pipeline construction, hard problems like barge-in and latency, engineering best practices, and capstone extensions.

← Back to Roadmaps /roadmaps Building Real-Time Voice Agents from Scratch - Learning Roadmap | Nemorize Loading roadmap... Learning Topics This roadmap covers the following topics: ✅ Part I: Foundations - ✅ Shape of a Voice Agent /roadmaps/building-real-time-voice-agents-from-scratch/lessons/019e6873-1262-7db5-9311-c80162b6688e - ⚪ mic → ASR → LLM → TTS Loop - ⚪ Trade Matrix - ✅ Audio Fundamentals /roadmaps/building-real-time-voice-agents-from-scratch/lessons/019e6873-1262-752c-a717-ef08fc8f6f0b - ⚪ SR IN vs SR OUT - ⚪ float32 ↔ int16 Conversions - ✅ VAD: Detecting Speech /roadmaps/building-real-time-voice-agents-from-scratch/lessons/019e6873-1262-7413-838c-53bc3384556b - ⚪ Threshold Tuning - ⚪ Pre-roll Buffer ✅ Part II: The Pipeline - ⚪ ASR with faster-whisper - ⚪ Model Size Trade-offs - ⚪ ASR as a Blocking Call - ⚪ LLM Streaming & State - ⚪ Speakable System Prompt - ⚪ The Commit Pattern - ⚪ TTS & Latency Trick - ⚪ pop sentences Deep Dive - ⚪ Kokoro vs Piper Backends ✅ Part III: The Hard Parts - ⚪ Barge-in: Interruption - ⚪ Yield-Point Latency - ⚪ Cancel Wire Protocol - ⚪ The Feedback Loop - ⚪ Browser AEC - ⚪ Playback State Machine - ⚪ Three Distinct Moments ✅ Part IV: Engineering It Well - ⚪ Frontend Audio Scheduling - ⚪ AudioWorklet for Mic Capture - ⚪ Gapless playHead Scheduling - ⚪ Concurrency & Orchestration - ⚪ run in executor Pattern - ⚪ asyncio vs Threads — Same Shape ✅ Part V: Make It Yours - ⚪ Capstone Extensions - ⚪ Measurable Latency Fork - ⚪ Extension Projects - ⚪ The Production Bridge - ⚪ Trade-offs You Now Own - ⚪ Why Hosted APIs Choose as They Do Community Feedback Share your thoughts and rate this roadmap Sign in to share your feedback and rate this roadmap Loading comments...