Browser Voice Interaction AI Pitfall Guide 2026 — 16 Common Traps with AEC, getUserMedia, and Headless Modes

📝 Originally published (in Japanese) at forge.workstyle.tech.

When building voice-based AI interactions in the browser (avatars, voice bots, streaming AI), you’ll inevitably hit pitfalls stemming from audio physics and browser implementation quirks. This article compiles 16 traps I encountered during product development, organized in a symptom → cause → solution lookup format. No need to read from top to bottom—jump straight to the symptom you’re facing.

#

Echo and Self-Response Issues

1. Avatar Responds to Its Own Voice (Despite echoCancellation: true

) #

Symptom: TTS audio is picked up by the mic, and STT recognizes it as user speech, creating a self-response loop. #

Cause: AEC (Acoustic Echo Cancellation) requires a reference signal (the "sound to cancel"). Only the browser's official playback paths (<audio>

/ WebRTC receiver tracks) serve as references. Custom playback via Web Audio API does not reliably function as a reference. #

Solution: Return TTS audio from the server as a WebRTC remote track and play it via an <audio>

element. This eliminates echoes without text-matching workarounds (tested: 99 seconds of continuous speech with speakers on, zero false user turn detections).

2. Echoes Are Gone, but Speaking Simultaneously with the Avatar Distorts My Voice and Causes Misrecognition

Symptom: Only during dual speech, proper nouns get mangled (e.g., "社員数" → "シャインズ"), especially at word beginnings. #

Cause: Fundamental AEC trade-off. To cancel echoes, AEC suppresses/distorts near-end (user) audio during dual speech. #

Solution: Mitigate in three layers: ① Increase mic Opus bitrate and enable FEC (see Pitfall 13) ② Provide vocabulary hints to STT (see separate article: use "recent avatar speech" as initial_prompt

, not a dictionary) ③ Instruct LLM: "Input is STT transcription with potential errors. Interpret unnatural words as phonetically similar terms and add confirmation prompts."

3. Can’t Suppress Audio from Other Apps (Music, Videos)

Symptom: Audio/lyrics from a YouTube video opened by the agent keep getting transcribed by STT. #

Cause: Browser AEC can only reference audio played by the same tab/app. Audio from other processes is indistinguishable from human speech to the mic. #

Solution: No technical silver bullet. Combine OS speaker separation (e.g., macOS "Voice Isolation"; request via voiceIsolation: true

—ignored on unsupported systems), headphones, and downstream noise rejection (LLM/tool-based filtering).

#

"Only the Beginning Is Unheard" Issues

4. Initial Speech at Session Start Isn’t Recognized

Symptom: First 10–20 seconds of speech go unanswered. Works normally afterward. #

Cause: Two factors: (a) AEC convergence—AEC only learns during actual playback, so unconverged dual speech suppresses the user's voice.

(b) autoGainControl ramp-up—Gain increases gradually, making initial speech too quiet. #

Solution: (a) Play short greetings/sound effects before the main session to train AEC ( screens are perfect for this).

(b) Set autoGainControl: false . If STT is server-side, volume fluctuations are handled well, and disabling AGC has minimal downsides. **Never disable **echoCancellation

.

5. Can’t Diagnose "No Response" Issues

Symptom: Unclear whether the issue is mic off, preprocessing loss, or server-side—leads to guesswork. #

Solution: Add two observability points: ① Server-side logging of input audio energy/probability (10Hz. Distinguishes "complete silence" from "distorted audio"). ② Level meter in UI for the actual transmitted stream (createMediaStreamSource

to read the same stream as transmission—doesn’t affect transmission).

6. Starting Mic Track with enabled=false

Causes "No Response" Complaints

Symptom: Implementation follows spec ("mic off until button press"), but users expect "always listening." #

Cause: Not a bug—** mismatch between design contract and user experience expectations**. A silent track looks indistinguishable from "quiet room" to the server. #

Solution: Align defaults with product promises. If "always listening" is a selling point, auto-enable on initialization completion (not during connection—that breaks mid-init conversations). Keep the button as a mute toggle.

#

Chromium Implementation Pitfalls

7. Remote Audio Connected to WebAudio Remains Silent (Lip Sync Doesn’t Work)

Symptom: WebRTC receiver stream connected to createMediaStreamSource

for analysis/processing, but no data flows. #

Cause: **Chromium doesn’t send remote **MediaStream

to WebAudio until a media element starts consuming it (long-standing behavior). #

Solution: Attach srcObject

to a muted

<audio>

element and call play() . The goal is to trigger playback, not produce sound.

8. MediaStreamAudioSourceNode

Silently Dies

Symptom: Input to processing graph stops after working for a while. No errors. #

Cause: Chrome may garbage collect unreferenced nodes, silently stopping input. #

Solution: Always retain references to nodes (and the above <audio>

element) via ref

or similar.

9. Volume Slider Doesn’t Work (New Path Only)

Symptom: GainNode

worked for local audio but stops working after switching to remote track playback. #

Cause: <audio> playback bypasses Web Audio GainNodes. #

Solution: Manipulate audioElement.volume

/ .muted

. During transition periods, apply changes to both paths.

#

Headless Environment (Streaming/Automation) Pitfalls

10. Headless Chromium Connection Dies at Startup

Symptom: Server never receives connection offer (offer

). #

Cause: Initial getUserMedia

throws an exception in headless environments (no mic). #

Solution: For mic-less use cases (streaming renderers), send a synthesized silent track via WebAudio. SDP and server pipelines behave identically to real mics.

11. MediaStreamAudioDestinationNode

Without Input May Stop

Symptom: Silent track connection works initially but RTP stops after a while, causing disconnection. If connected to MediaRecorder

, the muxer halts, stopping video too. #

Cause: Some implementations stop frame generation for destinations without input sources. #

Solution: As above, connect a ConstantSource(0) to keep rendering active.

12. Autoplay Policy Blocks Playback and AudioContext Startup

Symptom: No sound in headless mode / AudioContext

remains suspended

. #

**Solution**: Launch flag `--autoplay-policy=no-user-gesture-required`

+ `ctx.resume()`

. In regular browsers, always include a user interaction-triggered resume()

to "unlock" audio.

#

Quality and Tuning Pitfalls

13. Default WebRTC Mic Bitrate Is Surprisingly Low

Symptom: Switching from WS+raw PCM to WebRTC (Opus) reduced STT accuracy. #

Cause: Default Opus bitrate is ~30kbps. Lossy compression artifacts become critical under limiting conditions like dual speech. #

Solution: Modify answer

SDP fmtp

line to `maxaveragebitrate=128000;useinbandfec=1`

(answer-side `fmtp`

controls sender encoder). FEC also helps with packet loss over TURN.

14. VAD Silence Wait Dominates Response Latency

Symptom: Perceived 2-second response delay. Profiling shows neither LLM nor TTS is the bottleneck. #

**Cause**: Silence wait for end-of-speech detection (`stop_secs`

) accounts for >1 second. Shortening this causes mid-sentence cuts (another failure mode).

Solution: No silver bullet. Run STT concurrently during speech to preempt finalization, use turn detection models, and prioritize barge-in speed (instant interruption) for better perception.

15. "False Regression" in STT/TTS Latency Right After Deployment

Symptom: Post-release tests fail. Wasted time debugging code. #

Cause: Model lazy /prewarming (e.g., serial TTS speaker model loads) monopolizes GPU/event loop, causing inference APIs to wait tens of seconds. #

Solution: Wait for prewarming completion before testing (check log counts for "load start" vs "complete"). Integrate warmup checks into E2E harnesses.

16. Server Pipeline Idle Timeouts

Symptom: Sessions auto-terminate after 5 minutes of silence in valid use cases (streaming, monitoring). #

Cause: Default pipeline framework timeouts (e.g., Pipecat's 300s idle timeout) assume conversational use. "5 minutes inactive = abandoned" logic triggers self-cancellation. #

Solution: Separate settings by workload (e.g., cancel_on_idle_timeout=False

for streaming routes). Treat framework defaults as potential hazards when use cases shift.

#

Checklist (Save for Later)

- [ ] TTS playback uses WebRTC remote track +

<audio>

(AEC reference path)
- [ ] Pre-convergence "training sounds" are played for AEC
- [ ]

autoGainControl

disabled / echoCancellation

enabled

• [ ] Input energy logging + transmit level meter implemented
• [ ] Remote streams consumed by muted <audio>

before WebAudio

- [ ] Node/element references retained (GC prevention)
- [ ] Headless paths use synthesized silent track +
`ConstantSource(0)`

- [ ] Opus bitrate/FEC configured
- [ ] Testing waits for prewarming completion
- [ ] Idle timeouts reviewed per workload

Hope this saves fellow travelers in this swamp at least an hour of debugging!

source & further reading

dev.to — original article What Actually Breaks When You Put AI Agents In Front Of Real Customers Your agent's 'secure' network policy was off unless you did four steps — so it was off An Agent on a Leash, or why my AI agent doesn't make business decisions