{"slug": "i-made-my-voice-agent-feel-faster-by-streaming-sentences-not-audio", "title": "I Made My Voice Agent Feel Faster by Streaming Sentences, Not Audio", "summary": "A developer improved the perceived speed of a local voice agent by streaming text sentences from the LLM backend and synthesizing speech sentence-by-sentence instead of waiting for the full response. By overlapping TTS synthesis with playback and reducing backend latency through disabling extended thinking and conditional search tool use, the developer achieved a 5x reduction in time-to-first-byte and a much faster user experience.", "body_md": "The annoying thing about voice agents is that “the model is fast” does not mean the experience is fast.\n\nI had a small voice assistant running on a local device, talking to a hosted chat backend. The actual LLM call was only one part of the wait. The full path looked more like this:\n\n`/chat`\n\ncallIf you wait for step 4 to finish before starting step 5, the user hears nothing until the entire reply is done. That feels dead, even when the backend is technically fine.\n\nSo I changed the contract. The hardware client now calls the chat endpoint with `stream_tts: true`\n\n:\n\n```\nresponse = self.session.post(\n    f\"{CHAT_API_BASE}/chat\",\n    json={\"message\": message, \"stream_tts\": True},\n    timeout=30,\n    stream=True,\n)\n```\n\nThe backend yields text chunks as they arrive from the model. The device keeps a small buffer, splits complete sentences, and starts synthesizing each sentence immediately:\n\n```\n_SENTENCE_BOUNDARY_RE = re.compile(r'(?<=[.!?])\\s+')\n\ndef split_complete_sentences(buffer: str) -> tuple[list[str], str]:\n    *sentences, remainder = _SENTENCE_BOUNDARY_RE.split(buffer)\n    return [s.strip() for s in sentences if s.strip()], remainder\n```\n\nThat is deliberately boring. Not phoneme streaming. Not a custom audio protocol. Just sentence-level pipelining.\n\nThe next useful bit was overlapping synthesis and playback. A single background worker waits for synthesized WAV files and plays them in order, while a one-worker `ThreadPoolExecutor`\n\nstarts rendering the next sentence as soon as it is complete.\n\n```\nfor sentence in sentences:\n    future = synth_executor.submit(self.tts_engine.synthesize, sentence)\n    playback_queue.put((sentence, future))\n```\n\nThat removed the worst gap: “sentence one finished playing, now start thinking about sentence two’s audio.” The hardware now does the obvious thing a human expects — keep talking.\n\nI also cut backend time-to-first-token by doing less. For this conversational path, I turned off extended model thinking:\n\n```\n_NO_THINKING = types.ThinkingConfig(thinking_budget=0)\n```\n\nAnd I stopped advertising Google Search on every request. The search tool is only added when the prompt smells like it needs current/external information. Most turns do not.\n\n```\nif self._needs_search(built_contents):\n    all_functions.append(self.google_search)\n```\n\nThe result was about a 5x cut in chat time-to-first-byte for the common path, plus a much better perceived response because speech starts before the full answer exists.\n\nThe lesson was not “stream everything.” It was smaller than that:\n\nVoice agents do not need heroic architecture to feel better. Sometimes the fix is a regex, a queue, and deleting the expensive defaults.", "url": "https://wpnews.pro/news/i-made-my-voice-agent-feel-faster-by-streaming-sentences-not-audio", "canonical_source": "https://dev.to/toddsullivan/i-made-my-voice-agent-feel-faster-by-streaming-sentences-not-audio-4jej", "published_at": "2026-07-14 08:05:39+00:00", "updated_at": "2026-07-14 08:29:20.279279+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "developer-tools", "natural-language-processing"], "entities": ["Google"], "alternates": {"html": "https://wpnews.pro/news/i-made-my-voice-agent-feel-faster-by-streaming-sentences-not-audio", "markdown": "https://wpnews.pro/news/i-made-my-voice-agent-feel-faster-by-streaming-sentences-not-audio.md", "text": "https://wpnews.pro/news/i-made-my-voice-agent-feel-faster-by-streaming-sentences-not-audio.txt", "jsonld": "https://wpnews.pro/news/i-made-my-voice-agent-feel-faster-by-streaming-sentences-not-audio.jsonld"}}