{"slug": "a-live-voice-assistant-on-your-own-machine-whisper-pocket-tts-and-your-local-llm", "title": "A live voice assistant on your own machine: Whisper, Pocket TTS and your local LLM", "summary": "Locally Uncensored, an open source AGPL-3.0 desktop app for running AI models locally with over 180,000 downloads, added a live voice assistant mode in version 3.0.6 that runs speech recognition (Whisper base.en), a 15-voice Pocket TTS engine and turn taking entirely on-device while routing answers to the user's existing chat model. Measured latency from end of question to start of answer was 1.6 to 3.1 seconds on a Windows RTX 3060 12 GB with a local 9B model and 1.4 to 2.0 seconds on an M5 Pro Mac, with the voice engine consuming roughly 1 to 1.3 GB of memory; the feature is English-only and in beta.", "body_md": "A live voice assistant works like this: you talk, it answers out loud, you cut in, it stops. Since 3.0.6, Locally Uncensored has a mode that works that way with a model on your own machine. Speech recognition, the voice and the turn taking all run locally, and the model that answers is whatever chat model you already use.\n\nIt is a beta and English only. This post covers how it is built, what it costs in memory and time, and where it falls short.\n\nLocally Uncensored is an open source desktop app (AGPL-3.0) for running AI models locally, with 180,000+ downloads.\n\n```\nSpeech recognition   Whisper base.en       local\nVoice                Pocket TTS            local, 15 voices\nTurn taking          the voice engine      local\nAnswer               your chat model       local, or any provider you set up\n```\n\nThe voice engine is one process. It holds the speech detection, the speech recognition and the voice, and it loads them when it starts, not when a session starts. Voice is hidden while the app's Cloud switch is on, because the hosted side has no voice service yet.\n\nOpen Voice in the main navigation. The first time, you get a setup screen that shows the download size, the space needed and your free space before anything is fetched, plus a list of what is downloaded and where it goes. One click on Install. On a Mac that came to about 0.7 GB in our measurement.\n\nPick one of fifteen voices there or later: Alba, Anna, Azelma, Charles, Eponine, Eve, Fantine, George, Jane, Javert, Marius, Mary, Michael, Paul or Vera. Changing the voice fetches one small file.\n\nClick Start. A quiet chord says it is listening. Then you talk, it answers, and you can interrupt by speaking. The session is saved as a normal conversation.\n\nMeasured from the end of the question to the start of the answer, with a local model:\n\n```\nWindows, RTX 3060 12 GB, local 9B model    1.6 to 3.1 s\nM5 Pro Mac                                  1.4 to 2.0 s\n```\n\nThe first session after setup or a restart is the slow one. In our Windows tests it took 23 to 63 seconds to start cold, against about one second warm. What you say during that wait is kept for up to a minute and answered once the session is up, so you can start talking right away.\n\n```\nVoice engine in memory    about 1 GB on our Windows machine\n                          1.2 to 1.3 GB on the M5 Pro Mac\n```\n\nA switch next to the title shows the number LU measured and unloads the engine with one click, together with the chat model if the voice session loaded it. Without the click, the engine shuts down five minutes after the last session.\n\nFour switches sit next to the title: Voice settings, Agent, Where it may work and Thinking.\n\nWith Agent on, the voice session gets the agent's tools with the same permissions. Local models get a shorter list (web, files, shell and the screen) so the first answer comes quickly. A tool that needs your approval waits for a click on Approve or Reject. A spoken yes does not approve a tool, because speech recognition can mishear and the microphone hears the whole room.\n\nWhere it may work defaults to Workspace only: files stay in the session's folder, commands ask first, no screen. Whole computer opens the rest of your user account and the screen, after one extra question. System folders and the folders that hold keys and passwords stay closed at both levels.\n\nThinking is off by default. Switched on, the thought shows in the transcript and is never read aloud. A thought is capped between 128 and 4096 tokens, 512 by default, and if it runs longer LU ends it and asks the model for a direct answer.\n\nWith Whole computer on, you can ask for a task on the screen and hear the result. That is computer control, a separate beta with its own rules. Every new chat asks \"Allow computer control in this chat?\" and waits for a click. LU says aloud that it waits. A spoken yes does not allow it, a spoken no or stop declines.\n\nDuring a task, yes or no answers an ordinary action, and saying \"stop\" or \"wait\" on its own pauses it. \"Wait, what is two plus two?\" is a question for the model, not a pause. In our Windows test, typing a note in Notepad by voice took 11 to 22 seconds from the end of the command to the spoken result, with a local 9B model.\n\nOn a Mac, computer control is in the app but not fully tested yet.\n\nA live voice mode where everything except the answer is a fixed local pipeline (Whisper base.en, Pocket TTS, a turn taking engine), and the answer is your own chat model. About 1 to 1.3 GB of memory, an answer that starts 1.4 to 3.1 seconds after you stop talking, interruptions that work best with a headset, and tools and screen control behind clicks rather than spoken yeses. English only, and in beta.\n\nThe code is at [github.com/PurpleDoubleD/locally-uncensored](https://github.com/PurpleDoubleD/locally-uncensored). The setup, the measurements and the limits are written up in [VOICE.md](https://github.com/PurpleDoubleD/locally-uncensored/blob/master/VOICE.md) in the repository.\n\n*AI-assisted: I wrote this with help from an AI model and checked the facts against the code and VOICE.md.*", "url": "https://wpnews.pro/news/a-live-voice-assistant-on-your-own-machine-whisper-pocket-tts-and-your-local-llm", "canonical_source": "https://dev.to/purpledoubled/a-live-voice-assistant-on-your-own-machine-whisper-pocket-tts-and-your-local-llm-4a0b", "published_at": "2026-10-20 15:13:56+00:00", "updated_at": "2026-10-20 15:19:26.924736+00:00", "lang": "en", "topics": ["ai-tools", "ai-products", "large-language-models", "ai-agents"], "entities": ["Locally Uncensored", "Whisper", "Pocket TTS", "RTX 3060", "M5 Pro Mac"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-live-voice-assistant-on-your-own-machine-whisper-pocket-tts-and-your-local-llm", "markdown": "https://wpnews.pro/news/a-live-voice-assistant-on-your-own-machine-whisper-pocket-tts-and-your-local-llm.md", "text": "https://wpnews.pro/news/a-live-voice-assistant-on-your-own-machine-whisper-pocket-tts-and-your-local-llm.txt", "jsonld": "https://wpnews.pro/news/a-live-voice-assistant-on-your-own-machine-whisper-pocket-tts-and-your-local-llm.jsonld"}}