A live voice assistant works like this: you talk, it answers out loud, you cut in, it stops. Since 3.0.6, Locally Uncensored has a mode that works that way with a model on your own machine. Speech recognition, the voice and the turn taking all run locally, and the model that answers is whatever chat model you already use.
It is a beta and English only. This post covers how it is built, what it costs in memory and time, and where it falls short.
Locally Uncensored is an open source desktop app (AGPL-3.0) for running AI models locally, with 180,000+ downloads.
Speech recognition Whisper base.en local
Voice Pocket TTS local, 15 voices
Turn taking the voice engine local
Answer your chat model local, or any provider you set up
The voice engine is one process. It holds the speech detection, the speech recognition and the voice, and it loads them when it starts, not when a session starts. Voice is hidden while the app's Cloud switch is on, because the hosted side has no voice service yet.
Open Voice in the main navigation. The first time, you get a setup screen that shows the download size, the space needed and your free space before anything is fetched, plus a list of what is downloaded and where it goes. One click on Install. On a Mac that came to about 0.7 GB in our measurement.
Pick one of fifteen voices there or later: Alba, Anna, Azelma, Charles, Eponine, Eve, Fantine, George, Jane, Javert, Marius, Mary, Michael, Paul or Vera. Changing the voice fetches one small file.
Click Start. A quiet chord says it is listening. Then you talk, it answers, and you can interrupt by speaking. The session is saved as a normal conversation.
Measured from the end of the question to the start of the answer, with a local model:
Windows, RTX 3060 12 GB, local 9B model 1.6 to 3.1 s
M5 Pro Mac 1.4 to 2.0 s
The first session after setup or a restart is the slow one. In our Windows tests it took 23 to 63 seconds to start cold, against about one second warm. What you say during that wait is kept for up to a minute and answered once the session is up, so you can start talking right away.
Voice engine in memory about 1 GB on our Windows machine
1.2 to 1.3 GB on the M5 Pro Mac
A switch next to the title shows the number LU measured and unloads the engine with one click, together with the chat model if the voice session loaded it. Without the click, the engine shuts down five minutes after the last session.
Four switches sit next to the title: Voice settings, Agent, Where it may work and Thinking.
With Agent on, the voice session gets the agent's tools with the same permissions. Local models get a shorter list (web, files, shell and the screen) so the first answer comes quickly. A tool that needs your approval waits for a click on Approve or Reject. A spoken yes does not approve a tool, because speech recognition can mishear and the microphone hears the whole room.
Where it may work defaults to Workspace only: files stay in the session's folder, commands ask first, no screen. Whole computer opens the rest of your user account and the screen, after one extra question. System folders and the folders that hold keys and passwords stay closed at both levels.
Thinking is off by default. Switched on, the thought shows in the transcript and is never read aloud. A thought is capped between 128 and 4096 tokens, 512 by default, and if it runs longer LU ends it and asks the model for a direct answer.
With Whole computer on, you can ask for a task on the screen and hear the result. That is computer control, a separate beta with its own rules. Every new chat asks "Allow computer control in this chat?" and waits for a click. LU says aloud that it waits. A spoken yes does not allow it, a spoken no or stop declines.
During a task, yes or no answers an ordinary action, and saying "stop" or "wait" on its own s it. "Wait, what is two plus two?" is a question for the model, not a . In our Windows test, typing a note in Notepad by voice took 11 to 22 seconds from the end of the command to the spoken result, with a local 9B model.
On a Mac, computer control is in the app but not fully tested yet.
A live voice mode where everything except the answer is a fixed local pipeline (Whisper base.en, Pocket TTS, a turn taking engine), and the answer is your own chat model. About 1 to 1.3 GB of memory, an answer that starts 1.4 to 3.1 seconds after you stop talking, interruptions that work best with a headset, and tools and screen control behind clicks rather than spoken yeses. English only, and in beta.
The code is at github.com/PurpleDoubleD/locally-uncensored. The setup, the measurements and the limits are written up in VOICE.md in the repository.
AI-assisted: I wrote this with help from an AI model and checked the facts against the code and VOICE.md.