Build a Real-Time Speech-to-Text Pipeline with Whisper and WebSockets

A developer tutorial by Priya Nair shows how to build a self-hosted real-time speech-to-text pipeline using faster-whisper 1.2.1 and websockets 17.1, streaming 16 kHz mono PCM audio from a browser microphone to a local Whisper server over WebSockets. The server re-transcribes a growing audio buffer once per second to emit gray interim captions and commits black final text every 15 seconds, with all audio processed locally and no data leaving the machine. The guide requires Python 3.11 or newer, Chrome, Edge, or Safari, and about 150 MB of disk for the base model, which runs on CPU with int8 compute.

Build a Real-Time Speech-to-Text Pipeline with Whisper and WebSockets Stream microphone audio to a self-hosted Whisper server over WebSockets and render live captions in the browser. Priya Nair https://sourcefeed.dev/u/priya nair What you'll build A browser page that streams your microphone to a self-hosted Whisper https://github.com/openai/whisper model over a WebSocket and shows live captions: gray interim text that updates about once a second, committed black text every 15 seconds. Everything runs on your machine. No audio leaves it. Prerequisites Verified against faster-whisper 1.2.1, websockets 17.1, and ctranslate2 4.8.2 on Python 3.13 macOS 15 . Linux and Windows work the same way. - Python 3.11 or newer websockets 17.x requires 3.11+ - Chrome, Edge, or Safari. Firefox can't resample mic input to a 16 kHz AudioContext; see Troubleshooting - ~150 MB of disk for the base model, downloaded from Hugging Face on first run - A CPU is enough for the base model. GPU needs CUDA 12 and cuDNN 9 Step 1: Install the server dependencies faster-whisper https://github.com/SYSTRAN/faster-whisper runs Whisper on CTranslate2, about 4x faster than the reference implementation, which is what makes once-a-second re-transcription workable on a laptop CPU. websockets https://websockets.readthedocs.io/ handles the socket side. mkdir whisper-live && cd whisper-live python3 -m venv venv && source venv/bin/activate pip install "faster-whisper==1.2.1" "websockets==17.1" Step 2: Write the transcription server The protocol is deliberately dumb: the client sends raw 16-bit little-endian PCM at 16 kHz as binary frames, the server replies with JSON. The server appends every frame to a buffer and, once a second, re-transcribes the whole buffer and pushes the result as a partial . When the buffer hits 15 seconds it commits the text as a final and starts fresh. Re-transcribing the growing buffer costs more than a true incremental decoder, but it self-corrects earlier words as context arrives and needs no alignment logic. Save this as server.py : python import asyncio import json import numpy as np from faster whisper import WhisperModel from websockets.asyncio.server import serve SAMPLE RATE = 16000 BYTES PER SECOND = SAMPLE RATE 2 16-bit mono PCM MAX WINDOW SECONDS = 15 model = WhisperModel "base", device="cpu", compute type="int8" def transcribe pcm: bytes - str: audio = np.frombuffer pcm, dtype=np.int16 .astype np.float32 / 32768.0 segments, = model.transcribe audio, language="en", beam size=1, vad filter=True, condition on previous text=False, return " ".join segment.text.strip for segment in segments async def handler websocket : buffer = bytearray async def worker : last len = 0 while True: await asyncio.sleep 1.0 if len buffer == last len or len buffer < BYTES PER SECOND: continue snapshot = bytes buffer last len = len snapshot text = await asyncio.to thread transcribe, snapshot if len snapshot = MAX WINDOW SECONDS BYTES PER SECOND: Drop only what was transcribed, keeping audio that arrived while the model was busy. del buffer : len snapshot last len = 0 await websocket.send json.dumps {"type": "final", "text": text} else: await websocket.send json.dumps {"type": "partial", "text": text} task = asyncio.create task worker try: async for message in websocket: if isinstance message, bytes : buffer.extend message finally: task.cancel async def main : async with serve handler, "localhost", 8765 as server: print "Listening on ws://localhost:8765" await server.serve forever if name == " main ": asyncio.run main Four choices matter here. model.transcribe accepts a float32 numpy array at 16 kHz directly, so there's no temp-file dance. asyncio.to thread keeps the CPU-bound decode off the event loop, so incoming frames keep landing in the buffer while the model runs. beam size=1 with condition on previous text=False trades a little accuracy for latency and stops the model hallucinating repeats on partial audio. vad filter=True runs Silero VAD first, so silence and keyboard noise don't get decoded into phantom words. Step 3: Build the browser client Creating the AudioContext at 16 kHz makes the browser resample the mic for you, so the worklet only has to convert float32 samples to int16. Save this as index.html : < doctype html <html lang="en" <head <meta charset="utf-8" <title Live transcription</title </head <body <button id="start" Start listening</button <p id="status" Idle</p <p id="committed" </p <p id="partial" style="color: 888" </p <script const status = document.getElementById "status" ; const committed = document.getElementById "committed" ; const partial = document.getElementById "partial" ; document.getElementById "start" .onclick = async = { const ws = new WebSocket "ws://localhost:8765" ; ws.onmessage = event = { const msg = JSON.parse event.data ; if msg.type === "final" { committed.textContent += msg.text + " "; partial.textContent = ""; } else { partial.textContent = msg.text; } }; await new Promise resolve = ws.onopen = resolve ; const stream = await navigator.mediaDevices.getUserMedia { audio: { channelCount: 1, echoCancellation: true, noiseSuppression: true }, } ; // 16 kHz context: the browser resamples the mic to Whisper's input rate. const context = new AudioContext { sampleRate: 16000 } ; await context.audioWorklet.addModule "pcm-processor.js" ; const source = context.createMediaStreamSource stream ; const node = new AudioWorkletNode context, "pcm-processor" ; node.port.onmessage = event = { if ws.readyState === WebSocket.OPEN ws.send event.data ; }; source.connect node ; node.connect context.destination ; // keeps the graph pulled; the node outputs silence status.textContent = "Listening..."; }; </script </body </html Step 4: Add the AudioWorklet processor An AudioWorklet https://developer.mozilla.org/en-US/docs/Web/API/AudioWorklet hands you audio in 128-frame chunks on the audio thread. Sending a WebSocket message every 8 ms would be wasteful, so the processor batches 4096 samples ~256 ms per message. Save this as pcm-processor.js next to index.html : class PCMProcessor extends AudioWorkletProcessor { constructor { super ; this.samples = ; this.chunkSize = 4096; // ~256 ms at 16 kHz per WebSocket message } process inputs { const input = inputs 0 0 ; if input return true; for let i = 0; i < input.length; i++ this.samples.push input i ; if this.samples.length = this.chunkSize { const pcm = new Int16Array this.samples.length ; for let i = 0; i < this.samples.length; i++ { const s = Math.max -1, Math.min 1, this.samples i ; pcm i = s < 0 ? s 0x8000 : s 0x7fff; } this.port.postMessage pcm.buffer, pcm.buffer ; this.samples = ; } return true; } } registerProcessor "pcm-processor", PCMProcessor ; Verify it works Start the server. The first run downloads the model, then you should see: Listening on ws://localhost:8765 Serve the client from a second terminal in the same directory. getUserMedia needs a secure context and file:// doesn't qualify, but http://localhost does: python3 -m http.server 8000 Open http://localhost:8000 , click Start listening , allow the microphone, and talk. Within a couple of seconds the gray line starts updating and grows as you speak. Streaming a test sentence through this exact pipeline produced this sequence of messages on the wire: {"type": "partial", "text": "Hello World "} {"type": "partial", "text": "Hello World, this is a live transcription test streaming"} {"type": "partial", "text": "Hello World, this is a live transcription test streaming audio over a web socket."} Notice the second partial extends and corrects the first. That's the re-transcription window doing its job. Troubleshooting - TypeError: handler missing 1 required positional argument: 'path' — you copied a handler signature from an old tutorial. The legacy API called handler websocket, path ; the current websockets.asyncio API passes only the connection. Delete the path parameter. - NotSupportedError: Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported. — Firefox. It won't resample a mic stream into a 16 kHz context Mozilla bug 1725336 . Use Chrome, Edge, or Safari, or create the context at the default rate and downsample inside the worklet before converting to int16. - Unable to load any of {libcudnn ops.so.9.1.0, libcudnn ops.so.9.1, libcudnn ops.so.9, libcudnn ops.so} — you switched to device="cuda" without cuDNN 9. Run pip install nvidia-cublas-cu12 nvidia-cudnn-cu12 and add both packages' lib directories to LD LIBRARY PATH , or stay on CPU with compute type="int8" . - OSError: Errno 48 error while attempting to bind on address '127.0.0.1', 8765 — a previous server instance is still holding the port errno 98 on Linux . Find it with lsof -nP -iTCP:8765 and kill it. Next steps The 15-second commit window is the crudest part of this design: text can flicker until it's committed. Production streaming systems solve that with the LocalAgreement policy, where a prefix is committed once two consecutive transcriptions agree on it. whisper streaming https://github.com/ufal/whisper streaming implements it on top of faster-whisper and is a natural next read. On the model side, swap base for distil-large-v3 or large-v3 with device="cuda", compute type="float16" for much better accuracy at similar latency. And before exposing this beyond localhost, put the socket behind TLS wss:// , because browsers block mixed-content WebSockets and mic capture on insecure origins anyway. Sources & further reading 1. faster-whisper 1.2.1 https://pypi.org/project/faster-whisper/ — pypi.org 2. SYSTRAN/faster-whisper https://github.com/SYSTRAN/faster-whisper — github.com 3. websockets 17.1 documentation https://websockets.readthedocs.io/en/stable/ — websockets.readthedocs.io 4. AudioContext constructor https://developer.mozilla.org/en-US/docs/Web/API/AudioContext/AudioContext — developer.mozilla.org 5. Error in AudioContext.createMediaStreamSource with custom sample rate https://bugzilla.mozilla.org/show bug.cgi?id=1725336 — bugzilla.mozilla.org Priya Nair https://sourcefeed.dev/u/priya nair · AI & Developer Experience Writer Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to. Discussion 1 the firefox limitation is frustrating but honestly self-hosting whisper kills any native solution anyway. you're stuck doing your own audio resampling in js either way if you want cross-browser support — the article just sidesteps it entirely rather than showing how to actually handle it. would've been more useful than just listing it as a known issue.