I Made My Voice Agent Feel Faster by Streaming Sentences, Not Audio

A developer improved the perceived speed of a local voice agent by streaming text sentences from the LLM backend and synthesizing speech sentence-by-sentence instead of waiting for the full response. By overlapping TTS synthesis with playback and reducing backend latency through disabling extended thinking and conditional search tool use, the developer achieved a 5x reduction in time-to-first-byte and a much faster user experience.

The annoying thing about voice agents is that “the model is fast” does not mean the experience is fast. I had a small voice assistant running on a local device, talking to a hosted chat backend. The actual LLM call was only one part of the wait. The full path looked more like this: /chat callIf you wait for step 4 to finish before starting step 5, the user hears nothing until the entire reply is done. That feels dead, even when the backend is technically fine. So I changed the contract. The hardware client now calls the chat endpoint with stream tts: true : response = self.session.post f"{CHAT API BASE}/chat", json={"message": message, "stream tts": True}, timeout=30, stream=True, The backend yields text chunks as they arrive from the model. The device keeps a small buffer, splits complete sentences, and starts synthesizing each sentence immediately: SENTENCE BOUNDARY RE = re.compile r' ?<= . ? \s+' def split complete sentences buffer: str - tuple list str , str : sentences, remainder = SENTENCE BOUNDARY RE.split buffer return s.strip for s in sentences if s.strip , remainder That is deliberately boring. Not phoneme streaming. Not a custom audio protocol. Just sentence-level pipelining. The next useful bit was overlapping synthesis and playback. A single background worker waits for synthesized WAV files and plays them in order, while a one-worker ThreadPoolExecutor starts rendering the next sentence as soon as it is complete. for sentence in sentences: future = synth executor.submit self.tts engine.synthesize, sentence playback queue.put sentence, future That removed the worst gap: “sentence one finished playing, now start thinking about sentence two’s audio.” The hardware now does the obvious thing a human expects — keep talking. I also cut backend time-to-first-token by doing less. For this conversational path, I turned off extended model thinking: NO THINKING = types.ThinkingConfig thinking budget=0 And I stopped advertising Google Search on every request. The search tool is only added when the prompt smells like it needs current/external information. Most turns do not. if self. needs search built contents : all functions.append self.google search The result was about a 5x cut in chat time-to-first-byte for the common path, plus a much better perceived response because speech starts before the full answer exists. The lesson was not “stream everything.” It was smaller than that: Voice agents do not need heroic architecture to feel better. Sometimes the fix is a regex, a queue, and deleting the expensive defaults.