{"slug": "why-websockets-break-telephony-voice-ai-building-a-60ms-barge-in-loop-on-raw-rtp", "title": "Why WebSockets Break Telephony Voice AI: Building a 60ms Barge-In Loop on Raw 8kHz RTP", "summary": "Developer Wang Guo open-sourced a telephony voice AI lab that bypasses JSON-over-WebSocket wrappers in favor of raw 8kHz RTP through FreeSWITCH, using SOXR VHQ resampling to 16kHz for SenseVoice and VAD, and a 256-frame outbound ring buffer with a 0.16ms clearAudio() hard flush to achieve a roughly 60ms barge-in loop. The design splits the pipeline into two asymmetric loops so interruption detection relies on acoustic VAD rather than waiting 200-400ms for an LLM or full-sentence transcript.", "body_md": "**Author**: Wang Guo ([@wangguode](https://dev.to/wangguode))\n\n**Source Lab**: [https://github.com/wangguode/ai-agent-voice/tree/main/01-voice-agent-rtp-bargein](https://github.com/wangguode/ai-agent-voice/tree/main/01-voice-agent-rtp-bargein)\n\n**Stack**: FreeSWITCH 8kHz Raw RTP / Pipecat Full-Duplex / SenseVoice / SOXR VHQ Resampling\n\nMost Voice AI tutorials work great over a clean laptop microphone in a quiet room using browser WebSockets. The moment you connect that same pipeline to real enterprise telephony (PSTN / SIP trunks / FreeSWITCH), three brutal physical realities hit at once:\n\n`8kHz` G.711 (`PCMU`/` PCMA`) or `L16` audio, while modern acoustic models (SenseVoice, Whisper, Silero VAD) expect `16kHz` PCM. Naïve linear interpolation introduces aliasing artifacts that tank ASR accuracy and cause false VAD triggers.\nIn our open-source lab ([`01-voice-agent-rtp-bargein`](https://github.com/wangguode/ai-agent-voice/tree/main/01-voice-agent-rtp-bargein)), we bypassed generic JSON-over-WebSocket wrappers and engineered a deterministic telephony loop:\n\n```\n[PSTN / SIP Caller (8kHz RTP, 20ms frames)]\n        │\n        ▼ (UDP Raw RTP)\n[FreeSWITCH Media Gateway]\n        │\n        ├──► [SOXR VHQ Upsampler (8kHz ➔ 16kHz)] ──► [SenseVoice / Acoustic VAD (~18ms)]\n        │                                                       │\n        │                                                       ▼ (Barge-In Triggered!)\n        └──◄ [256-Frame Outbound Ring Buffer] ◄──── [0.16ms clearAudio() Hard Flush]\n```\n\n`SOXR_VHQ`) for `8kHz <-> 16kHz` conversion preserves vocal formants and prevents high-frequency aliasing from poisoning the local VAD classifier.`clearAudio()`` 0.16ms`—producing immediate silence on the caller's phone.\nA common mistake in Voice AI is waiting for the LLM or full-sentence STT transcript to confirm *\"the user is interrupting\"*. Transcription requires context windows (often 200ms–400ms).\n\nBy splitting the pipeline into **two asymmetric loops**:\n\nAll reference code, architecture diagrams, and runnable barge-in experiments are open-sourced:", "url": "https://wpnews.pro/news/why-websockets-break-telephony-voice-ai-building-a-60ms-barge-in-loop-on-raw-rtp", "canonical_source": "https://dev.to/wangguode/why-websockets-break-telephony-voice-ai-building-a-60ms-barge-in-loop-on-raw-8khz-rtp-27o4", "published_at": "2026-11-16 01:47:17+00:00", "updated_at": "2026-11-16 01:50:55.294174+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "natural-language-processing", "developer-tools", "ai-infrastructure"], "entities": ["Wang Guo", "FreeSWITCH", "Pipecat", "SenseVoice", "SOXR", "Whisper", "Silero VAD"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/why-websockets-break-telephony-voice-ai-building-a-60ms-barge-in-loop-on-raw-rtp", "markdown": "https://wpnews.pro/news/why-websockets-break-telephony-voice-ai-building-a-60ms-barge-in-loop-on-raw-rtp.md", "text": "https://wpnews.pro/news/why-websockets-break-telephony-voice-ai-building-a-60ms-barge-in-loop-on-raw-rtp.txt", "jsonld": "https://wpnews.pro/news/why-websockets-break-telephony-voice-ai-building-a-60ms-barge-in-loop-on-raw-rtp.jsonld"}}