{"slug": "when-is-it-safe-to-open-the-microphone-building-a-realtime-voice-agent-on-twilio", "title": "When is it safe to open the microphone? Building a realtime voice agent on Twilio", "summary": "A developer building a realtime voice agent on Twilio Media Streams found that the hardest part is not the signal path but the state machine that gates the microphone to prevent the agent from talking to itself. The developer implemented a gate that opens the microphone only when all Twilio 'mark' frames are acknowledged, the speech queue is empty, no TTS stream is active, the LLM is not generating, and the call is not tearing down. The developer also recommends returning the reason for a closed gate to aid debugging in production.", "body_md": "Wiring up a phone agent looks like a weekend project. Twilio Media Streams gives you a WebSocket with raw audio, you push it into a streaming STT, you feed the transcript to an LLM, you stream the reply into a TTS and send the bytes back. A few hundred lines. It works on the first call.\n\nThen you listen to a recording and the agent is talking to itself.\n\n```\nAgent:  \"Hello, how can I help you?\"\nSTT:    \"hello how can i help you\"          ← its own voice\nLLM:    \"Sure! What can I help you with?\"\nSTT:    \"sure what can i help you with\"     ← and again\n```\n\nNobody said a word. The call is in a loop.\n\nThis post is about the part that took the real time — not the signal path, but the state machine sitting on top of it. I run this in production on a German phone line, and every rule below exists because something broke on a real call.\n\nA phone line is not a mixing desk. There is one channel, and your own output comes back into it: through the caller's speaker, through network echo, through the conference bridge on the other end. Your STT does not know which words came from a human and which are your own TTS coming home.\n\nSo you need a gate. While the agent speaks, the microphone is closed and incoming transcripts are discarded. When the agent finishes, it reopens.\n\nThe whole difficulty is in the word *finishes*.\n\nThe first instinct is to close the microphone when TTS starts and reopen it when the TTS stream ends.\n\nThis is wrong, and it's wrong in a way that hides from you.\n\nThe end of your TTS stream is not the moment the caller hears the sentence. Between the last audio chunk you send and playback at the caller's ear sit the telephony platform's buffers and the network: anywhere from a couple of hundred milliseconds to well over a second, depending on the connection.\n\nRelease on stream end and the microphone opens **while the caller is still hearing your voice**. That's the feedback loop, right there.\n\nAnd here's the part that costs you a day: it never reproduces locally. On your machine the latency is a few milliseconds, so the window never opens wide enough to matter. It only shows up on a real call, over a real mobile network, ideally on the worst connection your caller has.\n\nTwilio supports a `mark`\n\nframe. You place one behind a block of audio, and Twilio sends it back to you when playback reaches that point.\n\nThat's the only honest signal you have. Not \"I finished sending\" — **\"they finished hearing.\"**\n\nSo the rule becomes: the microphone opens when no unacknowledged mark is outstanding.\n\nThat is necessary but not sufficient. Four more conditions turned out to be load-bearing, each after a specific failure:\n\n| Condition | What happens if you ignore it |\n|---|---|\n| All marks acknowledged | Feedback — the caller is still hearing the agent |\n| Speech queue empty | Opens in the gap between two sentences |\n| No TTS stream active | Race condition when more audio is pushed |\n| LLM not generating | Opens while the next sentence is still forming |\n| Not tearing down the call | The closing sentence gets cut off |\n\nOne design note that paid for itself many times over: the release check returns the **reason** alongside the decision, not a bare boolean. When something goes wrong on a live call, you get `false: marks`\n\nor `false: queue`\n\nin the log instead of a silent `False`\n\n, and you know immediately which of the five it was. Field logs are the only debugger you have on a phone call.\n\nIf you wait for the complete LLM response before speaking, every reply opens with a pause as long as the entire generation. On a phone call that's unbearable — a second of dead air feels like the line dropped.\n\nSo the token stream gets cut at sentence boundaries, and each finished sentence goes straight into the speech queue. Output starts as soon as the *first* sentence is ready.\n\nThis is a clear win, and it's also why the mark logic has to handle sets rather than a single value: one response produces several marks, and you need **all** of them acknowledged, not just the last one to arrive.\n\nAn assistant you cannot interrupt is unusable. Humans interrupt each other constantly, and a caller who has to wait through a wrong answer will hang up.\n\nBut if you treat every incoming transcript as an interruption, you get an agent that never finishes a sentence — because some of those transcripts are its own voice.\n\nThree hurdles, in order:\n\n**1. The agent must be speaking.** Nothing to interrupt otherwise.\n\n**2. Minimum length.** \"yes\", \"mhm\", \"right\" are backchannel signals. Humans emit them constantly while listening; they mean *I'm still here*, not *stop talking*. Three words turned out to be a reasonable floor.\n\n**3. It must not be echo.** This is the hard one, because STT practically never returns your own output word-for-word. It arrives with words dropped, merged, or lightly mangled. So two criteria run in parallel:\n\nThe comparison base is a rolling window over roughly the last 120 words the agent spoke. The window **must** be bounded. Unbounded, it grows across the call until eventually every caller utterance overlaps something the agent said twenty turns ago, and the agent goes deaf. That one is a slow, quiet failure — it doesn't crash, it just stops listening halfway through a long call.\n\nHere's the one I'd have paid money to know in advance.\n\nOn barge-in you send Twilio a `clear`\n\nframe, which discards the buffered audio. Reasonable — the caller is talking, you don't want your queued sentences playing over them.\n\nBut playback now never reaches the marks you placed in that discarded audio. **The receipts never arrive.**\n\nIf you don't flush pending marks on barge-in, the release check waits for the rest of the call on confirmations that do not exist. The microphone stays shut. The agent never hears the caller again. The connection is up, the line is quiet, and the conversation is dead.\n\nWhat makes it nasty is the distance between cause and symptom. The barge-in itself works perfectly — the agent stops mid-sentence, exactly as designed. The failure surfaces seconds later and looks exactly like an STT outage. I spent an embarrassing amount of time reading Deepgram logs.\n\nRelated, in the same family: the cancel flag has to be set **before** the queue is drained. The other way round and the still-running LLM stream drops new sentences into the queue you just emptied, and the agent wakes up again after the interruption — while the caller is mid-sentence.\n\nThe single most useful structural choice: all of this lives in one module with no network I/O and no external dependencies. It takes state in, returns a decision and a reason.\n\nWhich means the entire state machine is testable without a phone line, without API keys, and without a network connection. There are 24 tests, and each one documents a failure that actually happened on a call — the test names describe the symptom, not the method.\n\nThat matters more here than in most projects. A state bug reproducible only on a live call costs several minutes and a phone connection per iteration, and you can only test it as fast as you can talk. Moving the logic out of the I/O layer turned a two-minute feedback loop into a two-second one.\n\nThe repository has an unedited recording of a live call: the caller interrupts mid-sentence, the agent stops, and further down the call the caller goes quiet long enough that the silence watchdog fires and the agent asks whether they're still there — then correctly discards the echo of its own question.\n\nCode, sequence diagrams for all three state flows, and the recording:\n\n[https://github.com/bokatechsystems/realtime-voice-agent](https://github.com/bokatechsystems/realtime-voice-agent)\n\nMIT licensed. The gate logic is provider-agnostic — it assumes only that your telephony platform emits some form of playback receipt.\n\nIf you've built something similar and solved the timing differently, I'd genuinely like to hear about it. This is one of those problems where every implementation seems to arrive at its own set of five conditions.", "url": "https://wpnews.pro/news/when-is-it-safe-to-open-the-microphone-building-a-realtime-voice-agent-on-twilio", "canonical_source": "https://dev.to/petersoos/when-is-it-safe-to-open-the-microphone-building-a-realtime-voice-agent-on-twilio-3ddo", "published_at": "2026-08-09 15:15:30+00:00", "updated_at": "2026-08-09 15:17:36.714482+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "developer-tools"], "entities": ["Twilio", "Twilio Media Streams"], "alternates": {"html": "https://wpnews.pro/news/when-is-it-safe-to-open-the-microphone-building-a-realtime-voice-agent-on-twilio", "markdown": "https://wpnews.pro/news/when-is-it-safe-to-open-the-microphone-building-a-realtime-voice-agent-on-twilio.md", "text": "https://wpnews.pro/news/when-is-it-safe-to-open-the-microphone-building-a-realtime-voice-agent-on-twilio.txt", "jsonld": "https://wpnews.pro/news/when-is-it-safe-to-open-the-microphone-building-a-realtime-voice-agent-on-twilio.jsonld"}}