{"slug": "show-hn-full-duplex-real-time-voice-assistant-in-the-browser", "title": "Show HN: Full-duplex real-time voice assistant in the browser", "summary": "A developer built a full-duplex real-time voice assistant that runs entirely in the browser on WebGPU, using jax-js, with all stages (speech, ASR, LLM, TTS, and optional vision) processed locally. The assistant, inspired by Thinking Machines' interaction model and OpenAI's GPT-Live, supports barge-in, adaptive endpointing, and continuation-merge, and includes features like typed conversation memory and two-tier tools for weather and facts.", "body_md": "A real-time, full-duplex voice assistant that runs **entirely in your browser**\non WebGPU, built with [jax-js](https://github.com/ekzhang/jax-js).\n\nEvery stage (speech → ASR → LLM → TTS → speech, plus optional vision) runs locally in the tab; nothing is sent to a server.\n\nIt's inspired by the Thinking Machines\n[interaction model](https://thinkingmachines.ai/blog/interaction-models/) and\n[GPT-Live](https://openai.com/index/introducing-gpt-live/), rebuilt as a\nsmall-model cascade that fits in a browser tab. The goal is a conversation\nthat *feels* live: you can interrupt it mid-sentence, pause mid-thought\nwithout losing your turn, and it keeps searching in the background while you\ntalk.\n\n| Stage | Model | Runs on |\n|---|---|---|\n| Ear (ASR) |\n|\n\n[Silero VAD v5](https://github.com/snakers4/silero-vad), ported to TypeScript[SmolLM2-360M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct)(fp16)[Kyutai Pocket TTS](https://huggingface.co/kyutai/pocket-tts-without-voice-cloning)+[Mimi](https://github.com/kyutai-labs/moshi)codec (fp16)[D-FINE small](https://huggingface.co/bukuroo/D-FINE-ONNX)(COCO-80)`@jax-js/onnx`\n\nEverything shares the single WebGPU device. The streaming ASR lane is paused while the assistant speaks so it doesn't contend with TTS for the GPU; barge-in is therefore energy-based (see below), and captions resume the moment the assistant stops.\n\n**Full-duplex micro-turns**: a ~150 ms tick loop drives a deterministic, priority-ordered policy: adaptive** barge-in**(talk over the assistant, including its tool narrations, and the audio cuts in ~300 ms; the threshold auto-calibrates to the echo floor of each reply), adaptive**endpointing**, and time-awareness timers. A watchdog force-recovers the session if a reply ever stalls, so it can't wedge.** Continuation-merge**: if the endpoint fires on a mid-thought pause and you resume speaking before the reply's first audio, the unheard reply is aborted and both halves are answered as one turn (\"append, don't restart\").**Learned turn signal**: a pure-TypeScript port of Silero VAD v5 scores P(speech) every 32 ms on the CPU and drives speech onset, silence tracking, and the phantom-turn guard: keyboard noise and ambient swells never even latch an utterance, near-silence never reaches Whisper (so no hallucinated \"thank you\"s), and quiet speech still passes. A repetition-degeneracy gate drops decoder loops on top.**Eye (vision)**: enabled by default for webcam context, with a pre-load toggle to skip its 42 MB model, camera access, and GPU residency. D-FINE runs low-priority object detection (it yields the GPU to audio), smooths the person count, and answers \"what do you see?\" / \"how many people?\" / \"tell me about the person\" directly from the measurements. Proactive interjections (stepped away, phone spotted, slouching) are best-effort rule heuristics. The webcam shows as a corner PiP with detection boxes.**Typed conversation memory**: bounded facts the user explicitly states (name, trip, pet, favorite, plans, relationships) are retained and injected only when relevant; exact recall bypasses small-model guessing.**Two-tier tools**: factual asks are delegated so the small on-device model isn't left guessing: weather (\"what's the weather in Tokyo\" →[open-meteo](https://open-meteo.com/), in °F/mph), facts (\"who is Ada Lovelace\" → Wikipedia), plus instant offline**calculator** and**clock/date**. Web lookups speak a holding line and fetch in the background, then answer on the next silence and render a card; the card clears when the conversation moves on, and the spoken answer can be interrupted like any reply.\n\nThe turn-latency floor is set by the single GPU, so the work went into cutting\nGPU cost per token/frame rather than overlapping stages (which a single device\ncan't do; see [ docs/BENCHMARKS.md](/sachinkesiraju/jax-realtime/blob/main/docs/BENCHMARKS.md) for the full\nmap-reduce campaign log, including the negative results):\n\n**Fused decode**: the LLM decode step is fused from dozens of per-layer jit dispatches into one, and Pocket TTS from ~11 into two, cutting the command-buffer submit overhead that dominated per-step cost (~22% each).**GPU top-k sampling**: the LLM samples from a device-side top-64 (one small readback) instead of transferring the full vocab logits every token, folded into the fused step's single dispatch.**Stable prefill shapes**: every turn has a different prompt length, which otherwise forces jax-js to compile new traces mid-conversation. Prompts use 256-token buckets, the common buckets are warmed during loading, and the KV cache has one fixed capacity, so long conversations never hit a multi-second recompile stall mid-turn.**Faster confidence-aware ASR**: timestamp-gate candidate reductions are reused for confidence scoring instead of scanning the vocabulary again. ASR runs 5–7% faster while preserving all 21 paired clean/quiet/distorted transcripts; low-confidence failures request a repeat before invoking the LLM.**Bounded history window**: the prompt is capped at 8 messages so it never leaves the warm prefill buckets: first-token latency stays flat (~500–610 ms) across long sessions instead of doubling around turn 6, and the worst turn of a 12-turn session sits within ~150 ms of the median.**Deterministic memory fast paths**: exact recall and bounded trip, pet, and activity follow-ups can answer in a few milliseconds without model generation.** Smaller download**: Whisper ships a per-row int8 build (73 MB instead of 144 MB), while the TTS checkpoint omits 35 MB of audio-encoder weights never used for synthesis. The SmolLM2 brain stays full fp16 for conversation quality.\n\nRuntime behaviour is tunable at `src/tunables.ts`\n\n(read live, so A/B\nexperiments don't need a rebuild).\n\n**The assistant used to interrupt itself.** The barge-in threshold calibrated its echo floor during the reply's first ticks, which fall in the silent LLM/TTS latency gap, so the floor was ambient-level and the assistant's own playback echo tripped it. Calibration now only counts ticks where the TTS output analyser confirms audio is actually playing.**Whisper invents \"Thank you.\" from silence.** Every Whisper size does this (silent-outro captions in its training data), and it decodes with*high*confidence, so no model swap or confidence gate catches it. The fix is signal-side: unvoiced audio never reaches the decoder, on every endpoint path, including barge-in continuations, which originally skipped the guard.**A mid-sentence TTS cut sounds like a full stop.** Each chunk is synthesized as a complete utterance, so flushing at a bare word boundary produces sentence-final falling intonation plus a pause in the middle of your sentence. Chunks now only split at real punctuation: a measured latency cost, paid for prosody.\n\n```\nnpm install\nnpm run dev\n```\n\nOpen [http://localhost:5173](http://localhost:5173) in a WebGPU-capable browser (Chrome/Edge on desktop,\nSafari 26+). Click **Load models**: ~1.0 GB on first run, cached in OPFS\nafterwards. The Eye is on by default and requests camera access; uncheck it\nbefore loading to skip it. Then press the orb once and just talk, hands-free.\nPress it again to end.\n\nWeights come from\n[Hugging Face](https://huggingface.co/sachink98/jax-realtime-weights): SmolLM2\nat full fp16 for conversation quality, Whisper as a per-row int8 build\ndequantized at load (it transcribes the bench suite identically to fp16).\n\nThe pipeline stages, from microphone to speaker:\n\n| Path | What's there |\n|---|---|\n`src/mic.ts` |\n16 kHz PCM capture via an AudioWorklet. |\n`src/asr/` |\nWhisper encoder/decoder and features. `streaming.ts` transcribes live (LocalAgreement-2: words lock in once two passes agree) and filters out the assistant's own voice. `vad.ts` is the Silero VAD v5 port. |\n`src/llm/smollm.ts` |\nSmolLM2-360M forward pass with a KV cache: one fused GPU dispatch per token, bucket-padded prefill so jit traces are reused across turns. |\n`src/memory.ts` |\nExtraction and deterministic recall of facts the user shared. |\n`src/tts/` |\nPocket TTS flow-matching LM + the\n`AudioContext` player. `src/sentence-split.ts` chunks reply text at clause boundaries for synthesis. |\n\n`src/vision/`\n\n`@jax-js/onnx`\n\n, webcam `VisionSession`\n\n, COCO labels, box-dedupe and person-count smoothing.`src/tools/tools.ts`\n\n[MIT](/sachinkesiraju/jax-realtime/blob/main/LICENSE). Model inference code is adapted from the\n[jax-js repository](https://github.com/ekzhang/jax-js/tree/main/website/src/routes)\nby Eric Zhang (MIT licensed); model weights remain under their respective\nlicenses.", "url": "https://wpnews.pro/news/show-hn-full-duplex-real-time-voice-assistant-in-the-browser", "canonical_source": "https://github.com/sachinkesiraju/jax-realtime", "published_at": "2026-08-19 06:30:59+00:00", "updated_at": "2026-08-19 06:41:45.282442+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "ai-tools", "generative-ai"], "entities": ["jax-js", "Silero VAD v5", "SmolLM2-360M-Instruct", "Kyutai Pocket TTS", "Mimi", "D-FINE", "open-meteo", "Wikipedia"], "alternates": {"html": "https://wpnews.pro/news/show-hn-full-duplex-real-time-voice-assistant-in-the-browser", "markdown": "https://wpnews.pro/news/show-hn-full-duplex-real-time-voice-assistant-in-the-browser.md", "text": "https://wpnews.pro/news/show-hn-full-duplex-real-time-voice-assistant-in-the-browser.txt", "jsonld": "https://wpnews.pro/news/show-hn-full-duplex-real-time-voice-assistant-in-the-browser.jsonld"}}