{"slug": "i-built-my-own-voice-agent-in-python-to-understand-how-it-works", "title": "I Built My Own Voice Agent in Python to Understand How It Works", "summary": "A developer tutorial demonstrates building a working voice agent in roughly 330 lines of Python plus tests, without any voice framework, using faster-whisper for local speech-to-text, Anthropic's Claude as the streaming LLM, and OpenAI's text-to-speech for output. The agent slices microphone input into 20 ms frames (320 samples at 16 kHz), detects speech by comparing each frame's RMS volume against a running background-noise estimate, and supports barge-in by clearing the playback buffer and cancelling the Claude reply when the user interrupts. The post argues the loop is made of ordinary parts — ears, transcription, brain, and mouth — and keeps all hardware access in a single audio_io.py file so the rest can be tested with fake audio.", "body_md": "Voice agents feel like magic until you take one apart. You talk, it pauses for a beat, it answers in a natural voice, and if you cut it off mid-sentence it stops and listens. Behind that is a small loop of ordinary parts, and none of them is exotic.\n\nIn this post we build one from scratch in about 330 lines of Python, plus tests, with no voice framework. By the end you'll have an agent that listens through your microphone, thinks with Claude, talks back through your speakers, and stops talking the moment you interrupt it.\n\n## The shape of a voice agent\n\nEvery voice agent is the same four parts in a loop:\n\n1. **Ears:** turn a stream of microphone audio into \"someone is talking\" and \"they're done\".\n2. **Transcription (STT):** turn what they said into text.\n3. **Brain (LLM):** decide what to say, ideally while streaming the answer.\n4. **Mouth (TTS):** turn the answer into audio and play it, and stop instantly if the person interrupts.\n\nHere is how the pieces connect:\n\n```\nflowchart TD\n    subgraph hear[\"Listening\"]\n        mic[\"Microphone, 20 ms frames\"] --> vad[\"Turn detector\"]\n        vad -->|\"turn finished\"| stt[\"Speech-to-text\"]\n    end\n    subgraph speak[\"Speaking\"]\n        llm[\"Claude, streaming\"] --> split[\"Sentence splitter\"]\n        split --> tts[\"Text-to-speech\"]\n        tts --> buf[\"Playback buffer\"]\n        buf --> spk[\"Speakers\"]\n    end\n    stt --> llm\n    vad -.->|\"barge-in: clear the buffer\"| buf\n    vad -.->|\"barge-in: cancel the reply\"| llm\n```\n\nAnd the files we'll write, one per idea:\n\n| File | Job | \n|---|---|\n| `vad.py` | Detect speech and decide when a turn starts and ends | \n| `providers.py` | Whisper (STT), Claude (LLM), OpenAI (TTS), each behind one small method | \n| `sentences.py` | Cut a streaming reply into speakable sentences | \n| `playback.py` | The buffer between the agent and the speakers | \n| `audio_io.py` | The only file that touches real audio hardware | \n| `agent.py` | The orchestration, including interruptions | \n| `main.py` | Wiring | \n\nKeeping the hardware in one small file matters later: everything else can be tested with fake audio.\n\n## Setup\n\nYou need Python 3.10 or newer, a microphone, **headphones** (I'll explain why below), and two API keys: one for Anthropic (the brain) and one for OpenAI (the voice). Speech-to-text runs locally, so it needs no key.\n\nCreate a folder with this layout. Every file is shown in full later in the post, in the order we build it:\n\n```\nvoice-agent/\n  requirements.txt\n  vad.py\n  sentences.py\n  playback.py\n  audio_io.py\n  providers.py\n  agent.py\n  main.py\n  test_voice_agent.py\n```\n\nStart with `requirements.txt`:\n\n```\nanthropic\nhttpx\nnumpy\nsounddevice\nfaster-whisper\npytest\n```\n\nThe versions are left unpinned on purpose. Once everything works on your machine, run `pip freeze` and keep the exact versions you tested with. Then install and set your keys:\n\n```\npython -m venv .venv && source .venv/bin/activate\npip install -r requirements.txt\nexport ANTHROPIC_API_KEY=your-key\nexport OPENAI_API_KEY=your-key\n```\n\nOn Linux you may also need PortAudio (`sudo apt install libportaudio2`). On macOS and Windows the `sounddevice` package bundles it. The first run downloads the Whisper model.\n\n## Steps 1 and 2: Hearing, and knowing when someone is done\n\nThe microphone gives us a continuous stream of numbers. We slice it into 20 ms frames (320 samples at 16 kHz) and ask a simple question of each frame: is this speech?\n\nA production system would use a trained model such as Silero VAD or WebRTC VAD. For learning, a much simpler idea is enough: speech is louder than the room. We compute the volume of each frame (its RMS) and compare it to a running estimate of the background noise. If a frame is three times louder than the noise, it counts as speech.\n\nThe noise estimate needs one trick. Quiet frames update it quickly, but loud frames only nudge it. That way a fan switching on is eventually absorbed into the background, while a few seconds of speech barely moves the baseline.\n\nA single loud frame proves nothing, so the `TurnDetector` sits on top and runs a small state machine:\n\n```\nstateDiagram-v2\n    direction LR\n    Idle: Idle, keeps the last 300 ms of audio\n    Speaking: Speaking, collects every frame\n    [*] --> Idle\n    Idle --> Speaking: 120 ms of speech, emit start\n    Speaking --> Idle: 700 ms of silence, emit end with the audio\n```\n\nThree numbers control it:\n\n- `start_ms=120`**:** speech must last 120 ms before we believe it. A door slam or a cough is shorter than that.\n- `end_ms=700`**:** we call the turn finished after 700 ms of silence. This is the most important dial in the whole system. Shorter feels snappy but cuts people off when they pause to think. Longer is polite but adds dead air to every single reply. Every voice agent picks a spot on this dial.\n- `preroll_ms=300`**:** we only know someone is talking 120 ms after they started, by which point the first word is already gone. So we keep a rolling 300 ms of recent audio and prepend it to the utterance.\n\n``` python\n# vad.py\nfrom collections import deque\n\nimport numpy as np\n\nSAMPLE_RATE = 16_000\nFRAME_MS = 20\nFRAME_SAMPLES = SAMPLE_RATE * FRAME_MS // 1000   # 320 samples per frame\nFRAME_BYTES = FRAME_SAMPLES * 2                  # 16-bit audio, 2 bytes per sample\n\nclass EnergyVAD:\n    \"\"\"Says 'speech' when a frame is clearly louder than the room's background noise.\"\"\"\n\n    def __init__(self, ratio=3.0, floor=150.0):\n        self.ratio = ratio    # how many times louder than the noise counts as speech\n        self.floor = floor    # never trigger below this volume (RMS, int16 units)\n        self.noise = 100.0    # running estimate of the background level\n        self.boost = 1.0      # raise it to make triggering harder (we use this while talking)\n\n    def is_speech(self, frame: bytes) -> bool:\n        x = np.frombuffer(frame, dtype=np.int16).astype(np.float32)\n        rms = float(np.sqrt(np.mean(x * x))) if x.size else 0.0\n        speech = rms > max(self.floor, self.noise * self.ratio) * self.boost\n        # Quiet frames update the noise estimate quickly. Loud frames only nudge it, so a\n        # fan that switches on is eventually treated as background instead of speech.\n        rate = 0.0002 if speech else 0.05\n        self.noise += rate * (rms - self.noise)\n        return speech\n\nclass TurnDetector:\n    \"\"\"Turns a stream of 20 ms frames into 'start' and 'end' (with the audio) events.\"\"\"\n\n    def __init__(self, vad, start_ms=120, end_ms=700, preroll_ms=300, max_s=20):\n        self.vad = vad\n        self.start_frames = start_ms // FRAME_MS    # speech needed before we believe it\n        self.end_frames = end_ms // FRAME_MS        # silence needed before we call it done\n        self.max_frames = max_s * 1000 // FRAME_MS\n        self.preroll = deque(maxlen=preroll_ms // FRAME_MS)\n        self._reset()\n\n    def _reset(self):\n        self.speaking = False\n        self.run = 0          # consecutive speech frames seen while idle\n        self.silence = 0      # consecutive silent frames seen while speaking\n        self.frames = []\n\n    def feed(self, frame: bytes):\n        speech = self.vad.is_speech(frame)\n\n        if not self.speaking:\n            self.preroll.append(frame)\n            self.run = self.run + 1 if speech else 0\n            if self.run >= self.start_frames:\n                self.speaking = True\n                self.frames = list(self.preroll)    # keep the lead-in so the first word is not clipped\n                return \"start\", None\n            return None\n\n        self.frames.append(frame)\n        self.silence = 0 if speech else self.silence + 1\n        if self.silence >= self.end_frames or len(self.frames) >= self.max_frames:\n            audio = b\"\".join(self.frames)\n            self.preroll.clear()\n            self._reset()\n            return \"end\", audio\n        return None\n```\n\n## Step 3: Three services behind three tiny interfaces\n\nEach external service gets one class with one method. That keeps the agent logic free of vendor details and makes every piece swappable.\n\n**Speech-to-text** is `faster-whisper`, a fast local implementation of OpenAI's Whisper. It receives one finished utterance and returns text. That is the simplest design, but transcription only starts once you stop talking. More on that at the end.\n\n**The brain** is Claude, streamed. We use a small, fast model because in a voice conversation latency matters more than depth. The system prompt is doing real work: everything the model writes will be read aloud, so it must avoid markdown, lists and emoji, and keep answers short. If you want smarter answers, swap in a larger model and accept a longer pause.\n\n**Text-to-speech** is one HTTP request per sentence that returns raw 24 kHz, 16-bit mono audio. Raw PCM means no decoding step, so the bytes can go straight to the speakers.\n\n``` python\n# providers.py\nimport os\n\nimport anthropic\nimport httpx\nimport numpy as np\nfrom faster_whisper import WhisperModel\n\nSYSTEM = (\n    \"You are a voice assistant in a live spoken conversation. \"\n    \"Answer in one to three short sentences of plain spoken English. \"\n    \"No markdown, lists, emoji or abbreviations, because everything you write is read aloud. \"\n    \"If the user interrupted you, just respond to what they said last.\"\n)\n\nclass WhisperSTT:\n    \"\"\"Local speech-to-text. Takes one finished utterance, returns its text.\"\"\"\n\n    def __init__(self, model=\"base.en\"):\n        self.model = WhisperModel(model, device=\"cpu\", compute_type=\"int8\")\n\n    def transcribe(self, pcm: bytes) -> str:\n        audio = np.frombuffer(pcm, dtype=np.int16).astype(np.float32) / 32768.0\n        segments, _ = self.model.transcribe(\n            audio, language=\"en\", beam_size=1, condition_on_previous_text=False\n        )\n        return \" \".join(s.text.strip() for s in segments).strip()\n\nclass ClaudeLLM:\n    \"\"\"Streams the reply as it is written, so we can start speaking before it is finished.\"\"\"\n\n    def __init__(self, model=\"claude-haiku-5-5\"):\n        self.client = anthropic.AsyncAnthropic()    # reads ANTHROPIC_API_KEY\n        self.model = model\n\n    async def stream(self, history):\n        async with self.client.messages.stream(\n            model=self.model, max_tokens=300, system=SYSTEM, messages=history\n        ) as stream:\n            async for text in stream.text_stream:\n                yield text\n\nclass OpenAITTS:\n    \"\"\"Text in, raw 24 kHz 16-bit mono audio out.\"\"\"\n\n    def __init__(self, voice=\"alloy\", model=\"gpt-4o-mini-tts\"):\n        self.voice, self.model = voice, model\n        self.http = httpx.AsyncClient(\n            timeout=30, headers={\"Authorization\": f\"Bearer {os.environ['OPENAI_API_KEY']}\"}\n        )\n\n    async def synth(self, text: str) -> bytes:\n        r = await self.http.post(\n            \"https://api.openai.com/v1/audio/speech\",\n            json={\"model\": self.model, \"voice\": self.voice, \"input\": text, \"response_format\": \"pcm\"},\n        )\n        r.raise_for_status()\n        return r.content\n```\n\n## Step 4: Speak sentence by sentence\n\nClaude streams its reply a few words at a time. We could wait for the whole reply before speaking, but that adds seconds. We could speak every word as it arrives, but TTS needs whole sentences to get the rhythm right. The compromise is to cut the stream into sentences and send each one to TTS the moment it is complete.\n\nThe payoff is overlap. While one sentence is playing, the next is already being synthesized, so the listener hears a continuous voice even though the reply is still being written:\n\n```\nsequenceDiagram\n    participant C as Claude\n    participant A as Agent\n    participant T as Text-to-speech\n    participant S as Speakers\n    C-->>A: sentence 1 complete\n    A->>T: synthesize sentence 1\n    T-->>A: audio 1\n    A->>S: queue audio 1, playback starts\n    C-->>A: sentence 2 complete\n    A->>T: synthesize sentence 2\n    Note over S: sentence 1 is still playing\n    T-->>A: audio 2\n    A->>S: queue audio 2 behind audio 1\n```\n\nThe splitter has two small details. A sentence ends at `.`, `!` or `?` followed by whitespace, so `3.5` stays in one piece. And it won't emit anything shorter than 15 characters on its own, because a lone \"Hi.\" makes for an awkward TTS request and sounds clipped, so it waits and joins the next sentence.\n\n``` python\n# sentences.py\nimport re\n\n# A sentence ends at . ! or ? (plus any closing quote or bracket) followed by whitespace.\n# Requiring the whitespace keeps \"3.5\" and \"example.com\" in one piece.\nSENTENCE_END = re.compile(r\"\"\"[.!?][\"')\\]]*\\s\"\"\")\n\nclass SentenceSplitter:\n    \"\"\"Turns a stream of text deltas into sentences that are ready to be spoken.\"\"\"\n\n    def __init__(self, min_chars=15):\n        self.buf = \"\"\n        self.min_chars = min_chars    # \"Hi.\" alone is a poor TTS request, so it waits for a friend\n\n    def feed(self, delta: str) -> list[str]:\n        self.buf += delta\n        out, pos = [], 0\n        while (m := SENTENCE_END.search(self.buf, pos)):\n            if len(self.buf[: m.end()].strip()) < self.min_chars:\n                pos = m.end()                       # too short, keep accumulating\n                continue\n            out.append(self.buf[: m.end()].strip())\n            self.buf, pos = self.buf[m.end():], 0\n        return out\n\n    def flush(self) -> list[str]:\n        rest, self.buf = self.buf.strip(), \"\"\n        return [rest] if rest else []\n```\n\n## Step 5: The buffer between the agent and the speakers\n\nHere is the part that makes interruption possible. The sound card works on its own thread and demands audio on a fixed schedule: every few milliseconds it asks for the next chunk. The agent works on the asyncio thread and produces audio in bursts, one sentence at a time. The `PlaybackBuffer` sits between them.\n\n``` php\nflowchart LR\n    agent[\"Agent, asyncio thread\"] -->|\"push a sentence\"| buf[(\"PlaybackBuffer\")]\n    buf -->|\"read, every few ms\"| card[\"Sound card, audio thread\"]\n    agent -.->|\"clear on barge-in\"| buf\n```\n\nIt has three jobs:\n\n- `read(n)` is called by the sound card and always returns exactly`n` bytes, padded with silence when there is nothing to play.\n- `clear()` is the barge-in button. It throws away everything that has not been played yet, so the speakers go quiet almost immediately, apart from whatever the sound card has already queued.\n- `take_spoken()` returns the sentences whose playback actually started, so the agent can remember what it really said rather than what it meant to say.\n\n``` python\n# playback.py\nimport threading\nfrom collections import deque\n\nclass PlaybackBuffer:\n    \"\"\"Audio the agent wants to say. The sound card drains it, the agent fills it.\"\"\"\n\n    def __init__(self):\n        self._lock = threading.Lock()\n        self._chunks = deque()    # each item: [sentence, pcm_bytes, read_position]\n        self._spoken = []         # sentences whose playback has started\n\n    def push(self, sentence: str, pcm: bytes):\n        pcm = pcm[: len(pcm) // 2 * 2]    # int16 samples must be whole\n        with self._lock:\n            self._chunks.append([sentence, pcm, 0])\n\n    def read(self, nbytes: int) -> bytes:\n        \"\"\"Called from the audio thread. Always returns exactly nbytes, padded with silence.\"\"\"\n        out = bytearray()\n        with self._lock:\n            while len(out) < nbytes and self._chunks:\n                chunk = self._chunks[0]\n                sentence, pcm, pos = chunk\n                if pos == 0:\n                    self._spoken.append(sentence)\n                take = pcm[pos : pos + nbytes - len(out)]\n                out += take\n                chunk[2] += len(take)\n                if chunk[2] >= len(pcm):\n                    self._chunks.popleft()\n        out += b\"\\x00\" * (nbytes - len(out))\n        return bytes(out)\n\n    def clear(self):\n        \"\"\"Barge-in: throw away everything that has not been played yet.\"\"\"\n        with self._lock:\n            self._chunks.clear()\n\n    @property\n    def busy(self) -> bool:\n        with self._lock:\n            return bool(self._chunks)\n\n    def take_spoken(self) -> list[str]:\n        with self._lock:\n            spoken, self._spoken = self._spoken, []\n            return spoken\n```\n\nBoth audio callbacks run on separate threads. The microphone callback hands frames to asyncio with `call_soon_threadsafe`, and the speaker callback just pulls from the buffer.\n\n``` python\n# audio_io.py\nimport asyncio\n\nimport sounddevice as sd\n\nfrom vad import FRAME_SAMPLES, SAMPLE_RATE\n\nTTS_RATE = 24_000\n\nasync def mic_frames():\n    \"\"\"Yields 20 ms chunks of microphone audio as raw int16 bytes.\"\"\"\n    loop = asyncio.get_running_loop()\n    q: asyncio.Queue[bytes] = asyncio.Queue()\n\n    def on_audio(indata, frames, time_info, status):    # runs on the audio thread\n        loop.call_soon_threadsafe(q.put_nowait, bytes(indata))\n\n    with sd.RawInputStream(samplerate=SAMPLE_RATE, channels=1, dtype=\"int16\",\n                           blocksize=FRAME_SAMPLES, callback=on_audio):\n        while True:\n            yield await q.get()\n\ndef speaker(buffer):\n    \"\"\"An output stream that plays whatever is in the PlaybackBuffer, and silence otherwise.\"\"\"\n    def fill(outdata, frames, time_info, status):       # runs on the audio thread\n        outdata[:] = buffer.read(frames * 2)\n\n    return sd.RawOutputStream(samplerate=TTS_RATE, channels=1, dtype=\"int16\",\n                              blocksize=1024, callback=fill)\n```\n\n## Step 6: The agent, and what happens when you interrupt it\n\nThis is where everything comes together. Read `run` first: it feeds each microphone frame to the turn detector and reacts to two events.\n\n- `end` means the person finished a turn. We start a new`turn` task: transcribe, stream Claude's reply, split into sentences, synthesize, and push the audio into the buffer.\n- `start` **while we are talking** means they have begun speaking over us. We call`interrupt()` , which clears the buffer first (silence is the priority) and then cancels the reply task.\n\nHere is one interruption from start to finish:\n\n```\nsequenceDiagram\n    autonumber\n    participant D as TurnDetector\n    participant A as Agent\n    participant C as Claude\n    participant B as PlaybackBuffer\n    participant S as Speakers\n    D->>A: end of turn, with audio\n    A->>C: transcript, stream a reply\n    C-->>A: sentences stream back\n    A->>B: push audio for each sentence\n    B->>S: sentence 1 starts playing\n    D->>A: start, you are talking over it\n    A->>B: clear()\n    A-xC: cancel, which closes the stream\n    Note over A: history keeps sentence 1 only\n    D->>A: end of turn, with the new audio\n    A->>C: new transcript, with that history\n```\n\nFour details are worth slowing down for.\n\n**The turn task stays alive while audio plays.** The loop at the end of `turn` waits for the buffer to drain. Without it, the task would finish as soon as the last sentence was synthesized, which is typically long before it finishes being spoken, and there would be nothing left to cancel when the user interrupts.\n\n**Cancellation reaches all the way to Claude.** Cancelling the task unwinds the `aclosing` block around the stream, which closes the connection, so you stop generating words nobody will hear.\n\n**The history records what was said, not what was planned.** In the `finally` block we append only the sentences that started playing. If Claude wrote three sentences and you cut it off during the first, the history contains one. Otherwise the model would believe you heard an explanation you never did, and its next answer would build on it.\n\nIf the interruption lands before any audio plays, no assistant message is stored, and `add_user` merges your next utterance into the previous user message so the roles keep alternating.\n\n**Our own voice can interrupt us.** While the agent talks, its voice leaks from the speakers into the microphone, and the detector can mistake it for you. `BARGE_IN_BOOST` raises the bar to 2.5 times the usual threshold while we're speaking.\n\nThat is a patch, not a fix, which is why the setup says to wear headphones. The real fix is acoustic echo cancellation, which browsers and phone stacks do for you.\n\n``` python\n# agent.py\nimport asyncio\nimport time\nfrom contextlib import aclosing\n\nfrom playback import PlaybackBuffer\nfrom sentences import SentenceSplitter\nfrom vad import EnergyVAD, TurnDetector\n\nBARGE_IN_BOOST = 2.5    # while we talk, the mic must hear someone 2.5x louder to count it as an interruption\n\nclass VoiceAgent:\n    def __init__(self, stt, llm, tts, buffer: PlaybackBuffer):\n        self.stt, self.llm, self.tts, self.buffer = stt, llm, tts, buffer\n        self.detector = TurnDetector(EnergyVAD())\n        self.history: list[dict] = []\n        self.task: asyncio.Task | None = None    # the reply currently being generated or spoken\n\n    async def run(self, frames):\n        async for frame in frames:\n            # Our own voice leaks into the mic, so be harder to impress while we are talking.\n            self.detector.vad.boost = BARGE_IN_BOOST if self.buffer.busy else 1.0\n            event = self.detector.feed(frame)\n            if event is None:\n                continue\n            kind, audio = event\n            if kind == \"start\" and self.buffer.busy:\n                await self.interrupt()                     # they started talking over us: stop now\n            elif kind == \"end\":\n                await self.interrupt()                     # drop any reply that is still in flight\n                self.task = asyncio.create_task(self.turn(audio, time.monotonic()))\n\n    async def interrupt(self):\n        self.buffer.clear()                                # silence first, the rest can follow\n        if self.task and not self.task.done():\n            self.task.cancel()\n            await asyncio.gather(self.task, return_exceptions=True)\n\n    async def turn(self, audio: bytes, t0: float):\n        marks = {}\n        try:\n            text = await asyncio.to_thread(self.stt.transcribe, audio)\n            marks[\"stt\"] = time.monotonic() - t0\n            if len(text) < 2:                              # noise, or Whisper imagining words in silence\n                return\n            self.add_user(text)\n            print(f\"you:   {text}\")\n\n            splitter = SentenceSplitter()\n            async with aclosing(self.llm.stream(list(self.history))) as deltas:\n                async for delta in deltas:\n                    marks.setdefault(\"first_token\", time.monotonic() - t0)\n                    for sentence in splitter.feed(delta):\n                        await self.say(sentence, marks, t0)\n            for sentence in splitter.flush():\n                await self.say(sentence, marks, t0)\n\n            while self.buffer.busy:                        # stay alive until the audio is out, so we can be cut off\n                await asyncio.sleep(0.02)\n        finally:\n            # Whether we finished or were interrupted, remember only what was actually said aloud.\n            spoken = self.buffer.take_spoken()\n            if spoken:\n                self.history.append({\"role\": \"assistant\", \"content\": \" \".join(spoken)})\n\n    async def say(self, sentence: str, marks: dict, t0: float):\n        pcm = await self.tts.synth(sentence)\n        if \"first_audio\" not in marks:\n            marks[\"first_audio\"] = time.monotonic() - t0\n            print(f\"time:  stt {marks['stt']:.2f}s, first token {marks['first_token']:.2f}s, \"\n                  f\"first audio {marks['first_audio']:.2f}s (counted from the end-of-turn decision)\")\n        self.buffer.push(sentence, pcm)\n        print(f\"agent: {sentence}\")\n\n    def add_user(self, text: str):\n        if self.history and self.history[-1][\"role\"] == \"user\":    # the last reply was never heard: merge\n            self.history[-1][\"content\"] += \" \" + text\n        else:\n            self.history.append({\"role\": \"user\", \"content\": text})\n```\n\n## Step 7: Run it\n\nThe entry point only wires the parts together:\n\n``` python\n# main.py\nimport asyncio\n\nfrom agent import VoiceAgent\nfrom audio_io import mic_frames, speaker\nfrom playback import PlaybackBuffer\nfrom providers import ClaudeLLM, OpenAITTS, WhisperSTT\n\nasync def main():\n    buffer = PlaybackBuffer()\n    agent = VoiceAgent(WhisperSTT(), ClaudeLLM(), OpenAITTS(), buffer)\n    with speaker(buffer):\n        print(\"Listening. Wear headphones. Ctrl-C to quit.\")\n        await agent.run(mic_frames())\n\nif __name__ == \"__main__\":\n    try:\n        asyncio.run(main())\n    except KeyboardInterrupt:\n        pass\npython main.py\n```\n\nSay something, and you'll see your transcript, the agent's sentences as they are queued for speech, and a timing line like this for every reply:\n\n```\nyou:   what is a socket\ntime:  stt 0.xxs, first token 0.xxs, first audio 0.xxs (counted from the end-of-turn decision)\nagent: A socket is one end of a network connection.\n```\n\nThe `x` digits depend on your machine and your providers, so I won't quote numbers. Two things to keep in mind when reading them:\n\n- The clock starts when the turn detector decides you're done, so add the 700 ms endpointing wait to get what you actually experience.\n- The first audio mark is when audio enters the buffer, a few milliseconds before you hear it.\n\nA common target for a conversation that feels natural is roughly one second from the end of your speech to the start of the reply. Use the three timings to see which stage is eating your budget.\n\n## Testing a voice agent without a microphone\n\nIt sounds like voice code can only be tested by talking to it. But the hard parts of a voice agent are timing and state, not audio: when does a turn start, what counts as an interruption, what does the history contain afterwards. All of that can be tested with synthetic audio and fakes for each service.\n\nHere is the complete test file. It imports only `vad.py`, `sentences.py`, `playback.py` and `agent.py`, never `providers.py` or `audio_io.py`, so it runs on a machine with no PortAudio, no API keys and no Whisper model.\n\n``` python\n# test_voice_agent.py\nimport asyncio\n\nimport numpy as np\n\nfrom agent import VoiceAgent\nfrom playback import PlaybackBuffer\nfrom sentences import SentenceSplitter\nfrom vad import FRAME_BYTES, FRAME_SAMPLES, SAMPLE_RATE, EnergyVAD, TurnDetector\n\n# ---------- helpers: synthetic audio ----------\n\ndef to_frames(x):\n    x = x.astype(np.int16)\n    return [x[i : i + FRAME_SAMPLES].tobytes() for i in range(0, len(x), FRAME_SAMPLES)]\n\ndef noise(n_frames, amp=30, seed=0):\n    rng = np.random.default_rng(seed)\n    return to_frames(rng.normal(0, amp, n_frames * FRAME_SAMPLES))\n\ndef tone(n_frames, amp=3000, hz=217.3):   # odd frequency, so no two frames are identical\n    t = np.arange(n_frames * FRAME_SAMPLES) / SAMPLE_RATE\n    return to_frames(amp * np.sin(2 * np.pi * hz * t))\n\ndef events(frames, detector=None):\n    detector = detector or TurnDetector(EnergyVAD())\n    out = []\n    for i, f in enumerate(frames):\n        e = detector.feed(f)\n        if e:\n            out.append((i, *e))\n    return out\n\n# ---------- turn detection ----------\n\ndef test_detects_start_and_end_of_a_turn():\n    speech = tone(50)\n    got = events(noise(25) + speech + noise(60, seed=1))\n    assert [kind for _, kind, _ in got] == [\"start\", \"end\"]\n    (start_i, _, _), (end_i, _, audio) = got\n    assert 25 + 5 <= start_i <= 25 + 7          # about 120 ms after the first loud frame\n    assert 75 + 33 <= end_i <= 75 + 36          # about 700 ms after the last loud frame\n    # The lead-in is kept, so the first word is not clipped: the first speech frame is in there.\n    assert audio.find(speech[0]) == 9 * FRAME_BYTES       # 9 frames of lead-in, then the first speech frame\n\ndef test_short_blip_is_ignored():\n    assert events(noise(25) + tone(3) + noise(50, seed=1)) == []\n\ndef test_steady_loud_room_stops_counting_as_speech():\n    frames = noise(1500, amp=400)               # a fan, 30 seconds of it\n    got = events(frames)\n    assert got                                   # it fools the detector at first...\n    assert all(i < 1000 for i, kind, _ in got if kind == \"start\")   # ...but it adapts\n\n# ---------- sentence splitting ----------\n\nTEXT = \"Sure thing. A socket is one end of a network connection. It has an address and a port! Does that help?\"\nEXPECTED = [\n    \"Sure thing. A socket is one end of a network connection.\",\n    \"It has an address and a port!\",\n    \"Does that help?\",\n]\n\ndef split_all(chunks):\n    s, out = SentenceSplitter(), []\n    for c in chunks:\n        out += s.feed(c)\n    return out + s.flush()\n\ndef test_sentences_do_not_depend_on_how_the_stream_is_chunked():\n    assert split_all([TEXT]) == EXPECTED\n    assert split_all(list(TEXT)) == EXPECTED                    # one character at a time\n    assert split_all([TEXT[i : i + 7] for i in range(0, len(TEXT), 7)]) == EXPECTED\n\ndef test_decimals_are_not_sentence_ends():\n    assert split_all([\"It is about 3.5 miles away. Really.\"]) == [\"It is about 3.5 miles away.\", \"Really.\"]\n\n# ---------- playback buffer ----------\n\ndef test_playback_buffer_reads_across_chunks_and_pads_with_silence():\n    b = PlaybackBuffer()\n    b.push(\"a\", b\"\\x01\\x00\" * 5)     # 10 bytes\n    b.push(\"b\", b\"\\x02\\x00\" * 3)     # 6 bytes\n    assert b.read(8) == b\"\\x01\\x00\" * 4\n    assert b.take_spoken() == [\"a\"]\n    assert b.read(8) == b\"\\x01\\x00\" + b\"\\x02\\x00\" * 3\n    assert b.take_spoken() == [\"b\"]\n    assert not b.busy\n    assert b.read(4) == b\"\\x00\" * 4\n\ndef test_clear_keeps_only_sentences_that_started_playing():\n    b = PlaybackBuffer()\n    b.push(\"a\", b\"\\x01\\x00\" * 5)\n    b.push(\"b\", b\"\\x02\\x00\" * 5)\n    b.read(2)\n    b.clear()\n    assert not b.busy\n    assert b.read(4) == b\"\\x00\" * 4\n    assert b.take_spoken() == [\"a\"]\n\n# ---------- the whole agent, no microphone, no network ----------\n\nclass FakeSTT:\n    def __init__(self, lines):\n        self.lines = list(lines)\n\n    def transcribe(self, pcm):\n        return self.lines.pop(0)\n\nclass FakeLLM:\n    def __init__(self, replies):\n        self.replies, self.seen = list(replies), []\n\n    async def stream(self, history):\n        self.seen.append([dict(m) for m in history])\n        for sentence in self.replies.pop(0):\n            for word in sentence.split():\n                yield word + \" \"\n                await asyncio.sleep(0)\n\nclass FakeTTS:\n    async def synth(self, text):\n        await asyncio.sleep(0)\n        return b\"\\x01\\x00\" * 24_000      # exactly one second of audio per sentence\n\ndef drive(script, stt_lines, replies):\n    \"\"\"Runs the agent against a scripted microphone. The 'sound card' plays 20 ms per frame.\"\"\"\n    buffer = PlaybackBuffer()\n    llm = FakeLLM(replies)\n    agent = VoiceAgent(FakeSTT(stt_lines), llm, FakeTTS(), buffer)\n\n    async def pace(frames):\n        for f in frames:\n            buffer.read(960)                 # 20 ms of audio at 24 kHz\n            await asyncio.sleep(0.002)       # let the agent's tasks make progress\n            yield f\n\n    async def mic():\n        async for f in script(buffer, pace):\n            yield f\n\n    async def go():\n        await agent.run(mic())\n        while buffer.busy:                   # let the last reply finish playing\n            buffer.read(960)\n            await asyncio.sleep(0.001)\n        if agent.task:\n            await agent.task\n\n    asyncio.run(go())\n    return agent, llm\n\nS1 = \"A socket is one end of a network connection.\"\nS2 = \"You can think of it as a phone jack.\"\nS3 = \"Programs plug into it to talk.\"\nS4 = \"A port picks which program gets the data.\"\n\ndef test_barge_in_stops_the_agent_and_history_keeps_only_what_was_said():\n    async def script(buffer, pace):\n        async for f in pace(noise(25)): yield f\n        async for f in pace(tone(40)): yield f                       # \"what is a socket\"\n        while not buffer._spoken:                                    # wait for the agent to start talking\n            async for f in pace(noise(1)): yield f\n        async for f in pace(noise(15)): yield f                      # it talks for a moment\n        async for f in pace(tone(40)): yield f                       # \"no wait, what is a port\"\n        async for f in pace(noise(300, seed=2)): yield f\n\n    agent, llm = drive(script, [\"what is a socket\", \"no wait, what is a port\"], [[S1, S2, S3], [S4]])\n\n    assert len(llm.seen) == 2\n    assert llm.seen[0] == [{\"role\": \"user\", \"content\": \"what is a socket\"}]\n    assert llm.seen[1] == [                      # the model is told only what the user actually heard\n        {\"role\": \"user\", \"content\": \"what is a socket\"},\n        {\"role\": \"assistant\", \"content\": S1},\n        {\"role\": \"user\", \"content\": \"no wait, what is a port\"},\n    ]\n    assert agent.history[-1] == {\"role\": \"assistant\", \"content\": S4}\n\ndef test_uninterrupted_reply_is_spoken_in_full():\n    async def script(buffer, pace):\n        async for f in pace(noise(25)): yield f\n        async for f in pace(tone(40)): yield f\n        async for f in pace(noise(400, seed=2)): yield f\n\n    agent, _ = drive(script, [\"what is a socket\"], [[S1, S2]])\n    assert agent.history == [\n        {\"role\": \"user\", \"content\": \"what is a socket\"},\n        {\"role\": \"assistant\", \"content\": f\"{S1} {S2}\"},\n    ]\n\ndef test_empty_transcript_is_ignored():\n    async def script(buffer, pace):\n        async for f in pace(noise(25)): yield f\n        async for f in pace(tone(40)): yield f\n        async for f in pace(noise(100, seed=2)): yield f\n\n    agent, llm = drive(script, [\"\"], [])\n    assert llm.seen == [] and agent.history == []\n```\n\nRun it from the project folder:\n\n```\npytest\n```\n\nThe file has two halves worth reading closely.\n\n**Turn detection.** Frames of low noise and a sine tone stand in for speech. We check that turns start and end where they should, that the lead-in audio is kept, that a short blip is ignored, and that a steady loud room stops fooling the detector. The sentence splitter and the playback buffer get plain input and output checks.\n\n**The whole agent.** Speech-to-text, Claude and text-to-speech are replaced by fakes that return scripted text and one second of audio per sentence. A fake sound card reads 20 ms from the playback buffer for every microphone frame, so time is counted in frames instead of seconds and the scenario is deterministic.\n\nIn the barge-in test, the user asks a question, the agent starts speaking, and 15 frames later the user interrupts. We then check what the model is told on the next turn: the question, only the first sentence of the interrupted answer, and the interruption.\n\nTwo lessons from writing these:\n\n- **Break your code on purpose.** Remove the barge-in branch, record the whole planned reply, drop the pre-roll audio, and confirm a test fails each time. This exposed a weak pre-roll test: a tone at a round frequency repeats every few frames, so the first speech frame looked identical to a later one and the test passed even without the lead-in. An odd frequency such as 217.3 Hz fixed it.\n- **Know what the tests do not cover.** These tests cover the logic: turn detection, sentence splitting, the buffer, and the whole barge-in flow. They cannot cover your microphone, your speakers, or the three real services. Those you exercise by running the agent.\n\n## From empty folder to running agent\n\n1. Create the folder and `requirements.txt` from the Setup section, then install the dependencies.\n2. Copy the seven source files from Steps 1 to 7, plus `test_voice_agent.py` , into the folder.\n3. Run `pytest` . All ten tests should pass.\n4. Set the two API keys, put on headphones, and run `python main.py` . The first start downloads the Whisper model, so give it a moment.\n5. Say something, then interrupt the answer halfway through.\n\n## What this version gets wrong\n\nIt is a learning tool, and it has real limits:\n\n- **The detector is naive.** Volume is a weak proxy for speech. Music, a loud keyboard or a second voice will trigger it. A trained VAD fixes most of that.\n- **Transcription waits for you to finish.** Streaming STT, which transcribes while you speak, takes the whole transcription time off the critical path and is usually the biggest latency win available.\n- **TTS is one request per sentence.** Streaming the audio bytes as they are generated would cut the delay before the first word.\n- **Whisper imagines words in near-silence.** It sometimes returns a phantom \"Thank you.\" for background noise. The`len(text) < 2` check only catches the empty case, so a real system filters more aggressively.\n- **Errors are not handled.** An exception inside the turn task is easy to miss, because nobody awaits the task. Wrap`turn` in a try/except that logs and recovers, and retry failed API calls.\n- **The history grows forever.** Cap or summarize it for long conversations.\n- **There are no tools.** A real agent would look things up and take actions. That means handling tool-use blocks in the Claude stream, and deciding what the agent says while a tool runs.\n\n## When to use a framework instead\n\nBuilding it by hand shows where the time goes and why each design choice exists, but for production you would normally not keep it. LiveKit Agents and Pipecat handle transport, echo cancellation, trained turn-taking models and telephony. Speech-to-speech models skip the three-stage pipeline entirely, trading control over each stage for lower latency and more natural speech.\n\nHaving built the loop yourself, you'll know what those tools do for you, and which knob to reach for when something feels off: the pause, the cut-off, or the interruption that didn't land.", "url": "https://wpnews.pro/news/i-built-my-own-voice-agent-in-python-to-understand-how-it-works", "canonical_source": "https://zyvop.com/i-built-my-own-voice-agent-in-python-to-understand-how-it-works-uf1d9", "published_at": "2026-11-02 12:15:00+00:00", "updated_at": "2026-11-02 12:21:31.286853+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "developer-tools"], "entities": ["Claude", "Anthropic", "OpenAI", "Whisper", "faster-whisper", "Python", "Silero VAD", "WebRTC VAD"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-built-my-own-voice-agent-in-python-to-understand-how-it-works", "markdown": "https://wpnews.pro/news/i-built-my-own-voice-agent-in-python-to-understand-how-it-works.md", "text": "https://wpnews.pro/news/i-built-my-own-voice-agent-in-python-to-understand-how-it-works.txt", "jsonld": "https://wpnews.pro/news/i-built-my-own-voice-agent-in-python-to-understand-how-it-works.jsonld"}}