Show HN: Hushkey – Silent dictation hotkey for macOS A developer released Hushkey, a proof-of-concept macOS menu-bar app that performs silent dictation by reading lips, jaw and throat motion from the webcam at 30 fps instead of listening to audio, typing recognized text at the cursor in any app. The pipeline runs entirely locally — Apple Vision face landmarks feed a mouth and neck crop to HushNet, an Auto-VSR conformer lip reader kept frozen plus LoRA adapters and a new kinematic branch for lip aperture, jaw drop and throat motion, followed by beam search, an optional local LLM and paste-at-cursor. The developer states accuracy depends on the pretrained lip reader and on personalization, with a "Train on my mouth" feature recording roughly 40 silent prompts to train a few-MB personal adapter on the Mac, and a `hushkey eval --compare-…` command reporting paired-bootstrap p-values for whether the adapter or throat stream helps. Silent dictation for macOS. Hold a key, mouth your words with no sound, release, and the text appears at your cursor in any app. It works like Wispr Flow, except it reads your lips, jaw and throat from the webcam instead of listening. Status: proof of concept. The pipeline runs end to end: native Mac app → local model server → text at the cursor, with adapter training on your own clips. Accuracy depends on the pretrained lip reader and on how much you personalise it; see Honest expectations honest-expectations . Everything runs locally. No audio is used and no video leaves your Mac. hold Right-⌥ ──► webcam 30 fps ──► Apple Vision face landmarks ──► mouth + neck crop ──┐ │ localhost text typed at cursor ◄── optional local LLM ◄── beam search + LM ◄── HushNet ◄───┘ Auto-AVSR VSR + LoRA adapters + lip/jaw/throat kinematic branch | A native Mac app | Menu-bar app with webcam permission, push-to-talk on a modifier key, double-tap for hands-free, Esc to cancel, a non-activating HUD with a live mouth preview and partial transcript, and paste-at-cursor with clipboard restore. | | HushNet | The Auto-AVSR conformer lip reader, frozen , plus LoRA adapters and a new kinematic branch : lip aperture/width, jaw drop and their velocities, plus throat motion energy and vertical optical flow below the chin . It enters through a zero-initialised gate, so the model starts exactly as the pretrained one and only uses the extra stream if it lowers the loss. | | Silent-speech adaptation | Voiced corpora LRS3, VoxCeleb2 are pushed toward silent webcam speech with hypo-articulation, tempo and webcam augmentations. | | Personalisation in the app | "Train on my mouth" records ~40 silent prompts through the live pipeline and trains a few-MB personal adapter on your Mac. | | RL fine-tuning | Minimum-word-error-rate training, i.e. sequence-level policy gradient on beam-search N-best with reward = −word errors. | | Measurable claims | hushkey eval --compare-… gives a paired-bootstrap p-value: does the adapter help, does the throat stream help? | | repo | used for | not used, and why | |---|---|---| | mpc001/auto avsr https://github.com/mpc001/auto avsr | Backbone model and training recipe : conformer VSR, joint CTC/attention loss, mouth-ROI preprocessing, LRS3 SentencePiece units, frame-budget bucketing, adaptive time masking | Lightning trainer replaced by an adapter-only loop | | amanvirparhar/chaplin https://github.com/amanvirparhar/chaplin | Proved the real-time loop; Hugging Face mirror of the LRS3 V WER19.1 weights + RNN-LM; vendored ESPnet decoder; the 4-point eyes, nose, mouth alignment; LLM-cleanup idea | OpenCV window, pynput , MediaPipe detector: replaced by a native app with Apple Vision. Cleanup now also gets N-best, personal vocabulary and the target app | | joeblau/subvox https://github.com/joeblau/subvox | Mac-app structure for silent-speech prototypes; energy-gate segmentation with calibrated noise floor, pre-roll and hang-over MotionGate.swift ; the "is the signal real?" statistical test | AirPods head-motion input and DTW templates small-vocabulary ; listed as a roadmap channel | | facebookresearch/av hubert https://github.com/facebookresearch/av hubert | Considered as the backbone similar VSR accuracy | Non-commercial license and a fairseq fork that's painful on Apple Silicon. The adapter design would work on it too | | LipNet-PyTorch https://github.com/VIPL-Audio-Visual-Speech-Understanding/LipNet-PyTorch | Reference for CTC sentence lip reading and Conv3D front-ends | Trained on GRID's fixed 6-word grammar, so it can't do open dictation | | hilab-open-source/WatchYourMouth https://github.com/hilab-open-source/WatchYourMouth | Evidence that silent speech is decodable from geometry rather than RGB depth point clouds . It motivated the landmark-geometry kinematic branch and per-utterance normalisation | Needs a TrueDepth/depth sensor and CUDA PointNet++ ops; a depth channel is on the roadmap | Requirements: macOS 14+ Apple Silicon recommended . Hushkey uses Python 3.10–3.13 if you have it; otherwise it downloads a private copy for itself. About 2.5 GB of disk space. 1. Download Hushkey.dmg from the Releases https://github.com/amirrsyed/hushkey/releases page and drag Hushkey into Applications. 2. The first time, right-click Hushkey ▸ Open ▸ Open . The build isn't notarized by Apple, so a plain double-click is blocked once. If macOS says it's "damaged", run xattr -dr com.apple.quarantine /Applications/Hushkey.app . 3. In the welcome window, allow Camera and Accessibility , then click Install model server one-time, ≈1.5 GB, a few minutes . git clone https://github.com/amirrsyed/hushkey && cd hushkey xcode-select --install Apple's build tools, if you don't have them ./install.sh model server + app in /Applications, then opens it Re-run ./install.sh to update and ./uninstall.sh to remove. After installing, the source folder is no longer needed: everything lives in /Applications/Hushkey.app and ~/Library/Application Support/Hushkey . make lists the shortcuts make install , make dmg , make test , … . - Open at login : Hushkey ▸ Settings ▸ General. - If text shows in the overlay but isn't typed : macOS isn't letting Hushkey type. Open System Settings ▸ Privacy & Security ▸ Accessibility, remove Hushkey with – , add it again, and relaunch. Each new unsigned build counts as a new app. Until then Hushkey copies the text to your clipboard and tells you to press ⌘V. - Optional LLM cleanup as in Chaplin : install Ollama https://ollama.com , ollama pull qwen3:4b , and append --llm to the command in Settings ▸ Backend. make dev makes an editable backend in backend/.venv and points the app at it. make app builds the app only. hushkey serve --tiny runs a random tiny model for UI work without the 1 GB download the output is gibberish . CI builds the app and runs the tests. Pushing a tag such as v0.2.0 makes the Release workflow publish Hushkey.dmg . create an EMPTY repo named hushkey at https://github.com/new no README/license , then: cd hushkey git remote add origin https://github.com/amirrsyed/hushkey.git git push -u origin main git tag v0.1.0 && git push origin v0.1.0 optional: builds Hushkey.dmg into a GitHub Release | action | how | |---|---| | Dictate | hold Right ⌥ configurable: Right ⌘, fn , mouth silently, release | | Hands-free | double-tap the key; tap once more to finish | | Cancel | Esc while recording | | Auto-detect | menu ▸ Auto-detect mouthing : starts/stops on lip and jaw motion camera stays on | | Personalise | menu ▸ Train on my mouth… → record ~40 prompts → Train personal adapter | Tips: face the camera with your mouth and neck in view and even lighting. Exaggerate slightly, and dictate phrases rather than single words more context helps the decoder . The camera turns off 30 s after your last dictation unless auto-detect is on. See docs/TRAINING.md https://github.com/amirrsyed/hushkey/blob/main/docs/TRAINING.md . In short: hushkey prepare /data/lrs3/trainval data/lrs3 --layout lrs3 same preprocessing as live hushkey train adapt --manifest data/lrs3/manifest.jsonl → adapters/silent.pt GPU hushkey train personalize → adapters/personal.pt your enrollment clips hushkey train mwer RL fine-tune on word error rate hushkey eval