{"slug": "show-hn-hushkey-silent-dictation-hotkey-for-macos", "title": "Show HN: Hushkey – Silent dictation hotkey for macOS", "summary": "A developer released Hushkey, a proof-of-concept macOS menu-bar app that performs silent dictation by reading lips, jaw and throat motion from the webcam at 30 fps instead of listening to audio, typing recognized text at the cursor in any app. The pipeline runs entirely locally — Apple Vision face landmarks feed a mouth and neck crop to HushNet, an Auto-VSR conformer lip reader kept frozen plus LoRA adapters and a new kinematic branch for lip aperture, jaw drop and throat motion, followed by beam search, an optional local LLM and paste-at-cursor. The developer states accuracy depends on the pretrained lip reader and on personalization, with a \"Train on my mouth\" feature recording roughly 40 silent prompts to train a few-MB personal adapter on the Mac, and a `hushkey eval --compare-…` command reporting paired-bootstrap p-values for whether the adapter or throat stream helps.", "body_md": "**Silent dictation for macOS.** Hold a key, mouth your words with no sound, release, and the text\nappears at your cursor in any app. It works like Wispr Flow, except it reads your lips, jaw and throat\nfrom the webcam instead of listening.\n\n**Status: proof of concept.** The pipeline runs end to end: native Mac app → local model server →\ntext at the cursor, with adapter training on your own clips. Accuracy depends on the pretrained lip\nreader and on how much you personalise it; see [Honest expectations](#honest-expectations).\nEverything runs locally. No audio is used and no video leaves your Mac.\n\n```\n hold Right-⌥ ──► webcam 30 fps ──► Apple Vision face landmarks ──► mouth + neck crop ──┐\n                                                                                        │ localhost\n       text typed at cursor ◄── optional local LLM ◄── beam search + LM ◄── HushNet ◄───┘\n                                                                         (Auto-AVSR VSR\n                                                                          + LoRA adapters\n                                                                          + lip/jaw/throat\n                                                                            kinematic branch)\n```\n\n| **A native Mac app** | Menu-bar app with webcam permission, push-to-talk on a modifier key, double-tap for hands-free, Esc to cancel, a non-activating HUD with a live mouth preview and partial transcript, and paste-at-cursor with clipboard restore. | \n| **HushNet** | The Auto-AVSR conformer lip reader, **frozen** , plus LoRA adapters and a new**kinematic branch** : lip aperture/width, jaw drop and their velocities, plus**throat motion** (energy and vertical optical flow below the chin). It enters through a zero-initialised gate, so the model starts*exactly* as the pretrained one and only uses the extra stream if it lowers the loss. | \n| **Silent-speech adaptation** | Voiced corpora (LRS3, VoxCeleb2) are pushed toward silent webcam speech with hypo-articulation, tempo and webcam augmentations. | \n| **Personalisation in the app** | \"Train on my mouth\" records ~40 silent prompts through the live pipeline and trains a few-MB personal adapter on your Mac. | \n| **RL fine-tuning** | Minimum-word-error-rate training, i.e. sequence-level policy gradient on beam-search N-best with reward = −word errors. | \n| **Measurable claims** | `hushkey eval --compare-…` gives a paired-bootstrap p-value: does the adapter help, does the throat stream help? | \n\n| repo | used for | not used, and why | \n|---|---|---|\n| [mpc001/auto_avsr](https://github.com/mpc001/auto_avsr) | **Backbone model and training recipe** : conformer VSR, joint CTC/attention loss, mouth-ROI preprocessing, LRS3 SentencePiece units, frame-budget bucketing, adaptive time masking | Lightning trainer (replaced by an adapter-only loop) | \n| [amanvirparhar/chaplin](https://github.com/amanvirparhar/chaplin) | Proved the real-time loop; Hugging Face mirror of the `LRS3_V_WER19.1` weights + RNN-LM; vendored ESPnet decoder; the 4-point (eyes, nose, mouth) alignment; LLM-cleanup idea | OpenCV window, `pynput` , MediaPipe detector: replaced by a native app with Apple Vision. Cleanup now also gets N-best, personal vocabulary and the target app | \n| [joeblau/subvox](https://github.com/joeblau/subvox) | Mac-app structure for silent-speech prototypes; **energy-gate segmentation** with calibrated noise floor, pre-roll and hang-over (`MotionGate.swift` ); the*\"is the signal real?\"* statistical test | AirPods head-motion input and DTW templates (small-vocabulary); listed as a roadmap channel | \n| [facebookresearch/av_hubert](https://github.com/facebookresearch/av_hubert) | Considered as the backbone (similar VSR accuracy) | **Non-commercial license** and a fairseq fork that's painful on Apple Silicon. The adapter design would work on it too | \n| [LipNet-PyTorch](https://github.com/VIPL-Audio-Visual-Speech-Understanding/LipNet-PyTorch) | Reference for CTC sentence lip reading and Conv3D front-ends | Trained on GRID's fixed 6-word grammar, so it can't do open dictation | \n| [hilab-open-source/WatchYourMouth](https://github.com/hilab-open-source/WatchYourMouth) | Evidence that silent speech is decodable from geometry rather than RGB (depth point clouds). It motivated the landmark-geometry kinematic branch and per-utterance normalisation | Needs a TrueDepth/depth sensor and CUDA PointNet++ ops; a depth channel is on the roadmap | \n\nRequirements: macOS 14+ (Apple Silicon recommended). Hushkey uses Python 3.10–3.13 if you have it; otherwise it downloads a private copy for itself. About 2.5 GB of disk space.\n\n1. Download **Hushkey.dmg** from the[Releases](https://github.com/amirrsyed/hushkey/releases) page and drag Hushkey into Applications.\n2. The first time, **right-click Hushkey ▸ Open ▸ Open** . The build isn't notarized by Apple, so a plain\ndouble-click is blocked once. If macOS says it's \"damaged\", run`xattr -dr com.apple.quarantine /Applications/Hushkey.app` .\n3. In the welcome window, allow **Camera** and**Accessibility** , then click**Install model server** (one-time, ≈1.5 GB, a few minutes).\n\n```\ngit clone https://github.com/amirrsyed/hushkey && cd hushkey\nxcode-select --install     # Apple's build tools, if you don't have them\n./install.sh               # model server + app in /Applications, then opens it\n```\n\nRe-run `./install.sh` to update and `./uninstall.sh` to remove. After installing, the source folder is no\nlonger needed: everything lives in `/Applications/Hushkey.app` and `~/Library/Application Support/Hushkey`.\n`make` lists the shortcuts (`make install`, `make dmg`, `make test`, …).\n\n- **Open at login** : Hushkey ▸ Settings ▸ General.\n- **If text shows in the overlay but isn't typed** : macOS isn't letting Hushkey type. Open System Settings ▸\nPrivacy & Security ▸ Accessibility, remove Hushkey with**–** , add it again, and relaunch. Each new\nunsigned build counts as a new app. Until then Hushkey copies the text to your clipboard and tells you to press ⌘V.\n- **Optional LLM cleanup** (as in Chaplin): install[Ollama](https://ollama.com) ,`ollama pull qwen3:4b` ,\nand append`--llm` to the command in Settings ▸ Backend.\n\n`make dev` makes an editable backend in `backend/.venv` and points the app at it. `make app` builds the app\nonly. `hushkey serve --tiny` runs a random tiny model for UI work without the 1 GB download (the output is\ngibberish). CI builds the app and runs the tests. Pushing a tag such as `v0.2.0` makes the Release workflow\npublish `Hushkey.dmg`.\n\n```\n# create an EMPTY repo named hushkey at https://github.com/new (no README/license), then:\ncd hushkey\ngit remote add origin https://github.com/amirrsyed/hushkey.git\ngit push -u origin main\ngit tag v0.1.0 && git push origin v0.1.0     # optional: builds Hushkey.dmg into a GitHub Release\n```\n\n| action | how | \n|---|---|\n| Dictate | hold **Right ⌥** (configurable: Right ⌘, fn), mouth silently, release | \n| Hands-free | **double-tap** the key; tap once more to finish | \n| Cancel | **Esc** while recording | \n| Auto-detect | menu ▸ *Auto-detect mouthing* : starts/stops on lip and jaw motion (camera stays on) | \n| Personalise | menu ▸ *Train on my mouth…* → record ~40 prompts →*Train personal adapter* | \n\nTips: face the camera with your **mouth and neck in view** and even lighting. Exaggerate slightly, and\ndictate phrases rather than single words (more context helps the decoder). The camera turns off 30 s after\nyour last dictation unless auto-detect is on.\n\nSee [docs/TRAINING.md](https://github.com/amirrsyed/hushkey/blob/main/docs/TRAINING.md). In short:\n\n```\nhushkey prepare /data/lrs3/trainval data/lrs3 --layout lrs3      # same preprocessing as live\nhushkey train adapt --manifest data/lrs3/manifest.jsonl          # → adapters/silent.pt   (GPU)\nhushkey train personalize                                        # → adapters/personal.pt (your enrollment clips)\nhushkey train mwer                                               # RL fine-tune on word error rate\nhushkey eval <manifest> --adapter …/personal.pt --compare-adapter none\napp/                       Swift menu-bar app (SwiftPM package + xcodegen project.yml)\n  Sources/Hushkey/\n    App/                   HushkeyApp (scenes), AppModel (state machine)\n    Capture/               CameraService, CapturePipeline (pre-roll, streaming), MotionGate\n    Vision/                FaceKeypoints (Vision → 11 keypoints), FrameProcessor (crop + JPEG)\n    Backend/               BackendClient (WebSocket), BackendProcess (launch/train)\n    Input/                 HotkeyMonitor, TextInserter\n    UI/                    Menu, HUD, Settings, Onboarding, Enrollment\nbackend/\n  hushkey/\n    preprocess/            keypoints, mouth ROI (Auto-AVSR), throat motion, silent-speech augment\n    model/                 backbone loader, LoRA, HushNet, decoding, MWER, tokenizer\n    data/                  datasets + bucketing, MediaPipe prep for offline video\n    train.py evaluate.py   three training stages, WER + paired bootstrap\n    server.py engine.py    FastAPI WebSocket server, inference runtime\n    cleanup.py             casing/punctuation + optional Ollama LLM\n    _vendor/espnet         Apache-2.0 decoder/model code from Auto-AVSR/Chaplin (patched for py3.12+)\n  tests/                   synthetic talking faces → preprocessing, model, training, server\ndocs/                      ARCHITECTURE.md (protocol, schemas), TRAINING.md\nscripts/                   install_backend.sh (also bundled in the app), build_app.sh, package_dmg.sh, setup_backend.sh (dev)\ninstall.sh uninstall.sh    one-command install / removal; Makefile wraps everything\n.github/workflows/         ci.yml (tests, app build, installer smoke test), release.yml (tag → Hushkey.dmg)\n```\n\n- The base lip reader reports **19.1 % WER on LRS3** (voiced TED talks; Auto-AVSR's figure). Silent\nmouthing into a laptop webcam is harder: articulation is smaller and lighting worse. Expect clearly\nworse results until you've run`personalize` , and measure with`hushkey eval` .\n- Lip reading can't separate visually identical sounds (/p b m/, /f v/, /t d n/). The LM and the optional LLM cleanup resolve some of this from context.\n- Throat motion seen by an RGB webcam is a weak cue. It's gated, ablatable and measured, not assumed to help.\n- Not yet: streaming partials from a chunked encoder (current partials re-decode the buffer once a second), Core ML/ANE export, punctuation commands, multiple languages.\n\n- Developer ID signing + notarization in the release workflow: no right-click-to-open, and permissions survive updates\n- Core ML export of the frontend and encoder (run on the Neural Engine, drop the Python dependency for inference)\n- Chunked streaming encoder for word-by-word output\n- Extra channels behind the same gate: AirPods head/jaw motion (Subvox) and TrueDepth depth (Watch Your Mouth)\n- Voice-free commands (\"new line\", \"delete that\") via a small intent head on the CTC posteriors\n\nFrames are processed in memory by the local server and discarded after each utterance. The only exception\nis the enrollment clips you choose to record, which are saved as mouth crops plus keypoint features\n(not full frames) under `~/Library/Application Support/Hushkey/enrollment/`. Delete that folder to remove\nthem. The optional LLM runs through your local Ollama.\n\nHushkey code: MIT. Vendored ESPnet/Auto-AVSR code: Apache-2.0. Pretrained weights are **downloaded, not\nredistributed**, and carry the terms of their training data (research use). See [NOTICE](https://github.com/amirrsyed/hushkey/blob/main/NOTICE).\n\n- Ma et al., *Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels* , ICASSP 2023.\n- Petridis et al., *Visual-Only Recognition of Normal, Whispered and Silent Speech* , ICASSP 2018.\n- Prabhavalkar et al., *Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Models* , ICASSP 2018.\n- Hu et al., *LoRA: Low-Rank Adaptation of Large Language Models* , 2021.\n- Wang et al., *Watch Your Mouth: Silent Speech Recognition with Depth Sensing* , CHI 2024.\n- Shi et al., *Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction* (AV-HuBERT), ICLR 2022.", "url": "https://wpnews.pro/news/show-hn-hushkey-silent-dictation-hotkey-for-macos", "canonical_source": "https://github.com/amirrsyed/hushkey", "published_at": "2026-10-31 22:44:57+00:00", "updated_at": "2026-10-31 22:51:10.432238+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-tools", "computer-vision", "ai-products"], "entities": ["Hushkey", "macOS", "Apple Vision", "HushNet", "Auto-AVSR", "LRS3", "VoxCeleb2", "Wispr Flow"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-hushkey-silent-dictation-hotkey-for-macos", "markdown": "https://wpnews.pro/news/show-hn-hushkey-silent-dictation-hotkey-for-macos.md", "text": "https://wpnews.pro/news/show-hn-hushkey-silent-dictation-hotkey-for-macos.txt", "jsonld": "https://wpnews.pro/news/show-hn-hushkey-silent-dictation-hotkey-for-macos.jsonld"}}