# Build a Real-Time Speech-to-Text Pipeline with Whisper and WebSockets > Source: > Published: 2026-09-13 17:45:18+00:00 # Build a Real-Time Speech-to-Text Pipeline with Whisper and WebSockets Stream microphone audio to a self-hosted Whisper server over WebSockets and render live captions in the browser. [Priya Nair](https://sourcefeed.dev/u/priya_nair) ## What you'll build A browser page that streams your microphone to a self-hosted [Whisper](https://github.com/openai/whisper) model over a WebSocket and shows live captions: gray interim text that updates about once a second, committed black text every 15 seconds. Everything runs on your machine. No audio leaves it. ## Prerequisites Verified against faster-whisper 1.2.1, websockets 17.1, and ctranslate2 4.8.2 on Python 3.13 (macOS 15). Linux and Windows work the same way. - Python 3.11 or newer (websockets 17.x requires 3.11+) - Chrome, Edge, or Safari. Firefox can't resample mic input to a 16 kHz AudioContext; see Troubleshooting - ~150 MB of disk for the `base` model, downloaded from Hugging Face on first run - A CPU is enough for the `base` model. GPU needs CUDA 12 and cuDNN 9 ## Step 1: Install the server dependencies [faster-whisper](https://github.com/SYSTRAN/faster-whisper) runs Whisper on CTranslate2, about 4x faster than the reference implementation, which is what makes once-a-second re-transcription workable on a laptop CPU. [websockets](https://websockets.readthedocs.io/) handles the socket side. ``` mkdir whisper-live && cd whisper-live python3 -m venv venv && source venv/bin/activate pip install "faster-whisper==1.2.1" "websockets==17.1" ``` ## Step 2: Write the transcription server The protocol is deliberately dumb: the client sends raw 16-bit little-endian PCM at 16 kHz as binary frames, the server replies with JSON. The server appends every frame to a buffer and, once a second, re-transcribes the whole buffer and pushes the result as a `partial`. When the buffer hits 15 seconds it commits the text as a `final` and starts fresh. Re-transcribing the growing buffer costs more than a true incremental decoder, but it self-corrects earlier words as context arrives and needs no alignment logic. Save this as `server.py`: ``` python import asyncio import json import numpy as np from faster_whisper import WhisperModel from websockets.asyncio.server import serve SAMPLE_RATE = 16000 BYTES_PER_SECOND = SAMPLE_RATE * 2 # 16-bit mono PCM MAX_WINDOW_SECONDS = 15 model = WhisperModel("base", device="cpu", compute_type="int8") def transcribe(pcm: bytes) -> str: audio = np.frombuffer(pcm, dtype=np.int16).astype(np.float32) / 32768.0 segments, _ = model.transcribe( audio, language="en", beam_size=1, vad_filter=True, condition_on_previous_text=False, ) return " ".join(segment.text.strip() for segment in segments) async def handler(websocket): buffer = bytearray() async def worker(): last_len = 0 while True: await asyncio.sleep(1.0) if len(buffer) == last_len or len(buffer) < BYTES_PER_SECOND: continue snapshot = bytes(buffer) last_len = len(snapshot) text = await asyncio.to_thread(transcribe, snapshot) if len(snapshot) >= MAX_WINDOW_SECONDS * BYTES_PER_SECOND: # Drop only what was transcribed, keeping audio # that arrived while the model was busy. del buffer[: len(snapshot)] last_len = 0 await websocket.send(json.dumps({"type": "final", "text": text})) else: await websocket.send(json.dumps({"type": "partial", "text": text})) task = asyncio.create_task(worker()) try: async for message in websocket: if isinstance(message, bytes): buffer.extend(message) finally: task.cancel() async def main(): async with serve(handler, "localhost", 8765) as server: print("Listening on ws://localhost:8765") await server.serve_forever() if __name__ == "__main__": asyncio.run(main()) ``` Four choices matter here. `model.transcribe` accepts a float32 numpy array at 16 kHz directly, so there's no temp-file dance. `asyncio.to_thread` keeps the CPU-bound decode off the event loop, so incoming frames keep landing in the buffer while the model runs. `beam_size=1` with `condition_on_previous_text=False` trades a little accuracy for latency and stops the model hallucinating repeats on partial audio. `vad_filter=True` runs Silero VAD first, so silence and keyboard noise don't get decoded into phantom words. ## Step 3: Build the browser client Creating the `AudioContext` at 16 kHz makes the browser resample the mic for you, so the worklet only has to convert float32 samples to int16. Save this as `index.html`: ``` Live transcription

Idle

``` ## Step 4: Add the AudioWorklet processor An [AudioWorklet](https://developer.mozilla.org/en-US/docs/Web/API/AudioWorklet) hands you audio in 128-frame chunks on the audio thread. Sending a WebSocket message every 8 ms would be wasteful, so the processor batches 4096 samples (~256 ms) per message. Save this as `pcm-processor.js` next to `index.html`: ``` class PCMProcessor extends AudioWorkletProcessor { constructor() { super(); this.samples = []; this.chunkSize = 4096; // ~256 ms at 16 kHz per WebSocket message } process(inputs) { const input = inputs[0][0]; if (!input) return true; for (let i = 0; i < input.length; i++) this.samples.push(input[i]); if (this.samples.length >= this.chunkSize) { const pcm = new Int16Array(this.samples.length); for (let i = 0; i < this.samples.length; i++) { const s = Math.max(-1, Math.min(1, this.samples[i])); pcm[i] = s < 0 ? s * 0x8000 : s * 0x7fff; } this.port.postMessage(pcm.buffer, [pcm.buffer]); this.samples = []; } return true; } } registerProcessor("pcm-processor", PCMProcessor); ``` ## Verify it works Start the server. The first run downloads the model, then you should see: ``` Listening on ws://localhost:8765 ``` Serve the client from a second terminal in the same directory. `getUserMedia` needs a secure context and `file://` doesn't qualify, but `http://localhost` does: ``` python3 -m http.server 8000 ``` Open `http://localhost:8000`, click **Start listening**, allow the microphone, and talk. Within a couple of seconds the gray line starts updating and grows as you speak. Streaming a test sentence through this exact pipeline produced this sequence of messages on the wire: ``` {"type": "partial", "text": "Hello World!"} {"type": "partial", "text": "Hello World, this is a live transcription test streaming"} {"type": "partial", "text": "Hello World, this is a live transcription test streaming audio over a web socket."} ``` Notice the second partial extends and corrects the first. That's the re-transcription window doing its job. ## Troubleshooting - **`TypeError: handler() missing 1 required positional argument: 'path'`** — you copied a handler signature from an old tutorial. The legacy API called`handler(websocket, path)` ; the current`websockets.asyncio` API passes only the connection. Delete the`path` parameter. - **`NotSupportedError: Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported.`** — Firefox. It won't resample a mic stream into a 16 kHz context (Mozilla bug 1725336). Use Chrome, Edge, or Safari, or create the context at the default rate and downsample inside the worklet before converting to int16. - **`Unable to load any of {libcudnn_ops.so.9.1.0, libcudnn_ops.so.9.1, libcudnn_ops.so.9, libcudnn_ops.so}`** — you switched to` device="cuda"` without cuDNN 9. Run`pip install nvidia-cublas-cu12 nvidia-cudnn-cu12` and add both packages'`lib` directories to`LD_LIBRARY_PATH` , or stay on CPU with`compute_type="int8"` . - **`OSError: [Errno 48] error while attempting to bind on address ('127.0.0.1', 8765)`** — a previous server instance is still holding the port (errno 98 on Linux). Find it with`lsof -nP -iTCP:8765` and kill it. ## Next steps The 15-second commit window is the crudest part of this design: text can flicker until it's committed. Production streaming systems solve that with the LocalAgreement policy, where a prefix is committed once two consecutive transcriptions agree on it. [whisper_streaming](https://github.com/ufal/whisper_streaming) implements it on top of faster-whisper and is a natural next read. On the model side, swap `base` for `distil-large-v3` or `large-v3` with `device="cuda", compute_type="float16"` for much better accuracy at similar latency. And before exposing this beyond localhost, put the socket behind TLS (`wss://`), because browsers block mixed-content WebSockets and mic capture on insecure origins anyway. ## Sources & further reading 1. [faster-whisper 1.2.1](https://pypi.org/project/faster-whisper/) — pypi.org 2. [SYSTRAN/faster-whisper](https://github.com/SYSTRAN/faster-whisper) — github.com 3. [websockets 17.1 documentation](https://websockets.readthedocs.io/en/stable/) — websockets.readthedocs.io 4. [AudioContext() constructor](https://developer.mozilla.org/en-US/docs/Web/API/AudioContext/AudioContext) — developer.mozilla.org 5. [Error in AudioContext.createMediaStreamSource with custom sample rate](https://bugzilla.mozilla.org/show_bug.cgi?id=1725336) — bugzilla.mozilla.org [Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to. ## Discussion 1 the firefox limitation is frustrating but honestly self-hosting whisper kills any native solution anyway. you're stuck doing your own audio resampling in js either way if you want cross-browser support — the article just sidesteps it entirely rather than showing how to actually handle it. would've been more useful than just listing it as a known issue.