{"slug": "open-source-silent-speech-with-20-emg-sensors-and-a-webcam", "title": "Open-source silent speech with $20 EMG sensors and a webcam", "summary": "A team at Cal Hacks 13.0 built Silent Voice, an open-source system that converts silent mouthing into real-time speech in the user's cloned voice, using roughly $20 EMG sensors, a webcam, Claude and Cartesia Sonic 3.6; the project placed 2nd overall and won Cartesia's voice AI challenge. Silent Voice combines a live lip-reading model built on Auto-AVSR, an in-house PyTorch EMG model called SVR-1 pre-trained on Berkeley's silent speech data and fine-tuned on demo speaker Austin, and Claude via the Claude API for context and look-alike word repairs, locking in each word every 100 ms and speaking it through Cartesia before the sentence ends. The system targets laryngectomy patients who currently rely on electrolarynx buzzers, and shows \"Please repeat\" rather than guessing when it cannot resolve a word.", "body_md": "Silent Voice gives a voice back to people who lost their voice box. You mouth the words without a sound, and Silent Voice speaks them in your own voice while you're still mouthing, even on a phone call.\n\nWe built it at Cal Hacks 13.0, where it placed 2nd overall and won Cartesia's voice AI challenge.\nThe story of the weekend is in [Thai's writeup](https://thaimtl.github.io/cal-hacks-13/).\n\nA laryngectomy removes the voice box, usually after throat cancer. People keep their lips, tongue, and jaw, so the words are still there, only the sound is gone. Many of them speak through an electrolarynx, a handheld buzzer that sounds robotic and ties up a hand. We think they deserve a voice that sounds like them, keeps up with a real conversation, and works on an ordinary phone call, and that's what we're building.\n\nOur demo speaker is Austin, who's healthy, and no patients take part in the demo.\n\nSilent Voice has one mode, open mouth, no sound, in real time. Three readers work together, because each one sees what the others can't. Every 100 ms one joint search combines them and locks in each word as soon as they're confident enough.\n\n- The camera reads the lips and does most of the work. It also reads the facial expression, which sets the voice's emotion.\n- The EMG model reads the electrical signals of the face muscles, so it sees the tongue and jaw that no camera can. It settles words that look alike on the lips (\"bat\", \"mat\", \"pat\"), proposes its own guesses, and keeps going when the face is hard to see.\n- Claude follows the whole conversation, both sides. It predicts likely words before you start mouthing and repairs look-alike mistakes (\"bills\" → \"pills\" after a question about medicine). It only swaps look-alike words and never adds ideas.\n- Cartesia speaks each word in your cloned voice as soon as it locks, without waiting for the end of the sentence.\n- When the system can't tell what you said, it shows \"Please repeat\" instead of guessing.\n\nRead on for the details.\n\n- [docs/REALTIME_PIPELINE.md](https://github.com/thaimtl/Silent-Voice/blob/main/docs/REALTIME_PIPELINE.md) explains how the whole system works.\n- [docs/SVR1_SPEC.md](https://github.com/thaimtl/Silent-Voice/blob/main/docs/SVR1_SPEC.md) specifies the EMG model, SVR-1.\n- [docs/TEAM_TASKS_AND_INTERFACES.md](https://github.com/thaimtl/Silent-Voice/blob/main/docs/TEAM_TASKS_AND_INTERFACES.md) covers who builds what.\n- [HARDWARE.md](https://github.com/thaimtl/Silent-Voice/blob/main/HARDWARE.md) describes the rig, and[RECORDING.md](https://github.com/thaimtl/Silent-Voice/blob/main/RECORDING.md) the recording sessions.\n- [DEMO.md](https://github.com/thaimtl/Silent-Voice/blob/main/DEMO.md) runs the demo: one launcher starts everything in order (`Launch Silent Voice.bat` ).\n\nThe demo is two minutes, live, at our table.\n\n1. Austin calls the judge standing in front of us, through Twilio. The judge talks normally, and their words appear in the captions.\n2. Austin mouths his side of the conversation. His words appear as he mouths and play into the call in his cloned voice while he's still going.\n3. His facial expression sets the voice's emotion.\n4. The dashboard shows the real latency, which reader settled each word, and what Claude changed.\n\n| Part | Tool | \n|---|---|\n| Live lip reading | Ours ( `lipread/` ), built on the open[Auto-AVSR](https://github.com/mpc001/auto_avsr) model. We turned a clip-by-clip reader into a live one that streams word guesses as Austin mouths, knows when he starts and stops, and feeds every guess into one joint search with the EMG model and Claude. | \n| Face, mouth crop, expression | MediaPipe FaceLandmarker | \n| Muscle signals | BioAmp EXG Pill boards and an Arduino Uno, see [HARDWARE.md](https://github.com/thaimtl/Silent-Voice/blob/main/HARDWARE.md) | \n| EMG model | SVR-1, ours: PyTorch, pre-trained on Berkeley's silent speech data, fine-tuned on Austin | \n| Context and repairs | Claude, through the Claude API | \n| Voice clone and speech | Cartesia Sonic 3.6, Instant Voice Clone | \n| The other person's speech to text | Cartesia Ink | \n| Phone call | Twilio | \n| Compute | One laptop, RTX 5060 | \n\nAll code lives in this repo.\nDatasets, checkpoints, results, and Python environments live in a working directory outside it, and the code finds them through the environment variable `RECLAIM_WORKDIR` (datasets in `silent_speech/` and `data/`, plus `checkpoints/`, `benchmark_results/`, `recordings/`, `third_party/`).\nWithout it, the code expects data next to itself, which is how the cluster copy works.\nData folders are in `.gitignore`.\n\n| Folder | What it is | \n|---|---|\n| `svr/` | SVR-1, the EMG model: model, training, fine-tuning, preprocessing ( [docs/SVR1_SPEC.md](https://github.com/thaimtl/Silent-Voice/blob/main/docs/SVR1_SPEC.md) ) | \n| `fusion/` | The fusion process: the joint search over camera + EMG + Claude ( `search.py` ), the lock model (`lock.py` ), on the bus (`run.py` ), offline fitting and tests on the recordings (`offline.py` ), and its language model: a conversational 4-gram with a cache of the call (`lm.py` , built by`build_lm.py` ) | \n| `bench/` | Benchmarks and their frozen test splits | \n| `cluster/` | H100/H200 job scripts (PBS), smoke tests, runbook | \n| `camera/` | Camera-only baselines (Auto-AVSR, BRAVEn, LipLearner) | \n| `rig/` | EMG rig: Arduino firmware, serial reader, live hum filter, signal check, `emg.raw` publisher (`publish.py` ) | \n| `recorder/` | The recording app: camera + EMG + the session sentences ( [recorder/README.md](https://github.com/thaimtl/Silent-Voice/blob/main/recorder/README.md) ) | \n| `bus/` | The team bus: ZeroMQ wire format, the camera's topics ( `__init__.py` ) and every other endpoint and port (`team.py` ); a fake camera (`stub.py` ) and a typed stand-in for the judge (`type_judge.py` ) for tests | \n| `buslog/` | Logs every bus message to `$RECLAIM_WORKDIR/logs/` (`record.py` ) and replays a session (`replay.py` ) | \n| `claude_service/` | The Claude service: predicts Austin's next words and repairs look-alikes ( `brain.py` ), on the bus (`service.py` ) | \n| `dashboard/` | The live dashboard judges watch, from Thai's Figma v5, fed by the bus ( [dashboard/README.md](https://github.com/thaimtl/Silent-Voice/blob/main/dashboard/README.md) ) | \n| `demo/` | The demo launcher ( `launch.py` ,[DEMO.md](https://github.com/thaimtl/Silent-Voice/blob/main/DEMO.md) );`demo/dashboard/` : the captions dashboard prototype ([demo/dashboard/README.md](https://github.com/thaimtl/Silent-Voice/blob/main/demo/dashboard/README.md) ) | \n| `call/` | The phone call: Twilio, Austin's cloned voice (Cartesia Sonic) and the judge's captions (Cartesia Ink) ( [call/README.md](https://github.com/thaimtl/Silent-Voice/blob/main/call/README.md) ) | \n| `figures/` | The SVR-1 architecture figure, the SVR-1 vs Gaddy & Klein results ( `svr1_vs_gaddy_4ch*` ), and the scripts that draw them | \n| `tests/` | Smoke and unit tests | \n| `docs/` | Plans and specs, Markdown with PDF copies | \n\n- This is a prototype, not a medical device.\n- Lip reading gets about 1 in 5 words wrong even on normal speaking video, and silent mouthing is harder. The EMG model and Claude are there to close that gap.\n- The EMG model is trained for one person (Austin), from three recording sessions, see [RECORDING.md](https://github.com/thaimtl/Silent-Voice/blob/main/RECORDING.md) .\n- The voice runs a little behind the lips, because a word is spoken only once it's confident enough to lock.\n\n- [Digital Voicing of Silent Speech](https://aclanthology.org/2020.emnlp-main.445.pdf) , Gaddy and Klein, UC Berkeley, 2020, for the electrode sites and pre-training data.\n- [Auto-AVSR](https://github.com/mpc001/auto_avsr) , Ma et al., 2023, for lip reading.\n- [Lipflow](https://github.com/amywork777/lipflow) , MIT license, for live mouth cropping with MediaPipe.\n\nThai (EMG hardware and the Claude service), Austin (EMG model and fusion, and the speaker in the demo), Frank (Cartesia voice, the Twilio call, and the captions dashboard), and Evan (real-time lip reading and facial emotion).\n\nSilent Voice is open source under the [MIT license](https://github.com/thaimtl/Silent-Voice/blob/main/LICENSE).\nBundled third-party files keep their own licenses, which are Mermaid (MIT, `docs/vendor/mermaid.LICENSE`), the Liberation Sans and Atkinson Hyperlegible Next fonts (SIL Open Font License 1.1), and MediaPipe's face mesh data (Apache-2.0).", "url": "https://wpnews.pro/news/open-source-silent-speech-with-20-emg-sensors-and-a-webcam", "canonical_source": "https://github.com/thaimtl/Silent-Voice", "published_at": "2026-11-02 17:48:03+00:00", "updated_at": "2026-11-02 17:52:34.646772+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-products", "ai-tools", "natural-language-processing", "computer-vision"], "entities": ["Silent Voice", "Cal Hacks 13.0", "Cartesia", "Claude", "Anthropic", "Auto-AVSR", "MediaPipe FaceLandmarker", "Twilio"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/open-source-silent-speech-with-20-emg-sensors-and-a-webcam", "markdown": "https://wpnews.pro/news/open-source-silent-speech-with-20-emg-sensors-and-a-webcam.md", "text": "https://wpnews.pro/news/open-source-silent-speech-with-20-emg-sensors-and-a-webcam.txt", "jsonld": "https://wpnews.pro/news/open-source-silent-speech-with-20-emg-sensors-and-a-webcam.jsonld"}}