{"slug": "build-a-real-time-speech-to-text-pipeline-with-whisper-and-websockets", "title": "Build a Real-Time Speech-to-Text Pipeline with Whisper and WebSockets", "summary": "A developer tutorial by Priya Nair shows how to build a self-hosted real-time speech-to-text pipeline using faster-whisper 1.2.1 and websockets 17.1, streaming 16 kHz mono PCM audio from a browser microphone to a local Whisper server over WebSockets. The server re-transcribes a growing audio buffer once per second to emit gray interim captions and commits black final text every 15 seconds, with all audio processed locally and no data leaving the machine. The guide requires Python 3.11 or newer, Chrome, Edge, or Safari, and about 150 MB of disk for the base model, which runs on CPU with int8 compute.", "body_md": "# Build a Real-Time Speech-to-Text Pipeline with Whisper and WebSockets\n\nStream microphone audio to a self-hosted Whisper server over WebSockets and render live captions in the browser.\n\n[Priya Nair](https://sourcefeed.dev/u/priya_nair)\n\n## What you'll build\n\nA browser page that streams your microphone to a self-hosted [Whisper](https://github.com/openai/whisper) model over a WebSocket and shows live captions: gray interim text that updates about once a second, committed black text every 15 seconds. Everything runs on your machine. No audio leaves it.\n\n## Prerequisites\n\nVerified against faster-whisper 1.2.1, websockets 17.1, and ctranslate2 4.8.2 on Python 3.13 (macOS 15). Linux and Windows work the same way.\n\n- Python 3.11 or newer (websockets 17.x requires 3.11+)\n- Chrome, Edge, or Safari. Firefox can't resample mic input to a 16 kHz AudioContext; see Troubleshooting\n- ~150 MB of disk for the `base` model, downloaded from Hugging Face on first run\n- A CPU is enough for the `base` model. GPU needs CUDA 12 and cuDNN 9\n\n## Step 1: Install the server dependencies\n\n[faster-whisper](https://github.com/SYSTRAN/faster-whisper) runs Whisper on CTranslate2, about 4x faster than the reference implementation, which is what makes once-a-second re-transcription workable on a laptop CPU. [websockets](https://websockets.readthedocs.io/) handles the socket side.\n\n```\nmkdir whisper-live && cd whisper-live\npython3 -m venv venv && source venv/bin/activate\npip install \"faster-whisper==1.2.1\" \"websockets==17.1\"\n```\n\n## Step 2: Write the transcription server\n\nThe protocol is deliberately dumb: the client sends raw 16-bit little-endian PCM at 16 kHz as binary frames, the server replies with JSON. The server appends every frame to a buffer and, once a second, re-transcribes the whole buffer and pushes the result as a `partial`. When the buffer hits 15 seconds it commits the text as a `final` and starts fresh. Re-transcribing the growing buffer costs more than a true incremental decoder, but it self-corrects earlier words as context arrives and needs no alignment logic.\n\nSave this as `server.py`:\n\n``` python\nimport asyncio\nimport json\n\nimport numpy as np\nfrom faster_whisper import WhisperModel\nfrom websockets.asyncio.server import serve\n\nSAMPLE_RATE = 16000\nBYTES_PER_SECOND = SAMPLE_RATE * 2  # 16-bit mono PCM\nMAX_WINDOW_SECONDS = 15\n\nmodel = WhisperModel(\"base\", device=\"cpu\", compute_type=\"int8\")\n\ndef transcribe(pcm: bytes) -> str:\n    audio = np.frombuffer(pcm, dtype=np.int16).astype(np.float32) / 32768.0\n    segments, _ = model.transcribe(\n        audio,\n        language=\"en\",\n        beam_size=1,\n        vad_filter=True,\n        condition_on_previous_text=False,\n    )\n    return \" \".join(segment.text.strip() for segment in segments)\n\nasync def handler(websocket):\n    buffer = bytearray()\n\n    async def worker():\n        last_len = 0\n        while True:\n            await asyncio.sleep(1.0)\n            if len(buffer) == last_len or len(buffer) < BYTES_PER_SECOND:\n                continue\n            snapshot = bytes(buffer)\n            last_len = len(snapshot)\n            text = await asyncio.to_thread(transcribe, snapshot)\n            if len(snapshot) >= MAX_WINDOW_SECONDS * BYTES_PER_SECOND:\n                # Drop only what was transcribed, keeping audio\n                # that arrived while the model was busy.\n                del buffer[: len(snapshot)]\n                last_len = 0\n                await websocket.send(json.dumps({\"type\": \"final\", \"text\": text}))\n            else:\n                await websocket.send(json.dumps({\"type\": \"partial\", \"text\": text}))\n\n    task = asyncio.create_task(worker())\n    try:\n        async for message in websocket:\n            if isinstance(message, bytes):\n                buffer.extend(message)\n    finally:\n        task.cancel()\n\nasync def main():\n    async with serve(handler, \"localhost\", 8765) as server:\n        print(\"Listening on ws://localhost:8765\")\n        await server.serve_forever()\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\nFour choices matter here. `model.transcribe` accepts a float32 numpy array at 16 kHz directly, so there's no temp-file dance. `asyncio.to_thread` keeps the CPU-bound decode off the event loop, so incoming frames keep landing in the buffer while the model runs. `beam_size=1` with `condition_on_previous_text=False` trades a little accuracy for latency and stops the model hallucinating repeats on partial audio. `vad_filter=True` runs Silero VAD first, so silence and keyboard noise don't get decoded into phantom words.\n\n## Step 3: Build the browser client\n\nCreating the `AudioContext` at 16 kHz makes the browser resample the mic for you, so the worklet only has to convert float32 samples to int16. Save this as `index.html`:\n\n```\n<!doctype html>\n<html lang=\"en\">\n<head>\n  <meta charset=\"utf-8\">\n  <title>Live transcription</title>\n</head>\n<body>\n  <button id=\"start\">Start listening</button>\n  <p id=\"status\">Idle</p>\n  <p id=\"committed\"></p>\n  <p id=\"partial\" style=\"color: #888\"></p>\n  <script>\n    const status = document.getElementById(\"status\");\n    const committed = document.getElementById(\"committed\");\n    const partial = document.getElementById(\"partial\");\n\n    document.getElementById(\"start\").onclick = async () => {\n      const ws = new WebSocket(\"ws://localhost:8765\");\n      ws.onmessage = (event) => {\n        const msg = JSON.parse(event.data);\n        if (msg.type === \"final\") {\n          committed.textContent += msg.text + \" \";\n          partial.textContent = \"\";\n        } else {\n          partial.textContent = msg.text;\n        }\n      };\n      await new Promise((resolve) => (ws.onopen = resolve));\n\n      const stream = await navigator.mediaDevices.getUserMedia({\n        audio: { channelCount: 1, echoCancellation: true, noiseSuppression: true },\n      });\n      // 16 kHz context: the browser resamples the mic to Whisper's input rate.\n      const context = new AudioContext({ sampleRate: 16000 });\n      await context.audioWorklet.addModule(\"pcm-processor.js\");\n      const source = context.createMediaStreamSource(stream);\n      const node = new AudioWorkletNode(context, \"pcm-processor\");\n      node.port.onmessage = (event) => {\n        if (ws.readyState === WebSocket.OPEN) ws.send(event.data);\n      };\n      source.connect(node);\n      node.connect(context.destination); // keeps the graph pulled; the node outputs silence\n      status.textContent = \"Listening...\";\n    };\n  </script>\n</body>\n</html>\n```\n\n## Step 4: Add the AudioWorklet processor\n\nAn [AudioWorklet](https://developer.mozilla.org/en-US/docs/Web/API/AudioWorklet) hands you audio in 128-frame chunks on the audio thread. Sending a WebSocket message every 8 ms would be wasteful, so the processor batches 4096 samples (~256 ms) per message. Save this as `pcm-processor.js` next to `index.html`:\n\n```\nclass PCMProcessor extends AudioWorkletProcessor {\n  constructor() {\n    super();\n    this.samples = [];\n    this.chunkSize = 4096; // ~256 ms at 16 kHz per WebSocket message\n  }\n\n  process(inputs) {\n    const input = inputs[0][0];\n    if (!input) return true;\n    for (let i = 0; i < input.length; i++) this.samples.push(input[i]);\n    if (this.samples.length >= this.chunkSize) {\n      const pcm = new Int16Array(this.samples.length);\n      for (let i = 0; i < this.samples.length; i++) {\n        const s = Math.max(-1, Math.min(1, this.samples[i]));\n        pcm[i] = s < 0 ? s * 0x8000 : s * 0x7fff;\n      }\n      this.port.postMessage(pcm.buffer, [pcm.buffer]);\n      this.samples = [];\n    }\n    return true;\n  }\n}\n\nregisterProcessor(\"pcm-processor\", PCMProcessor);\n```\n\n## Verify it works\n\nStart the server. The first run downloads the model, then you should see:\n\n```\nListening on ws://localhost:8765\n```\n\nServe the client from a second terminal in the same directory. `getUserMedia` needs a secure context and `file://` doesn't qualify, but `http://localhost` does:\n\n```\npython3 -m http.server 8000\n```\n\nOpen `http://localhost:8000`, click **Start listening**, allow the microphone, and talk. Within a couple of seconds the gray line starts updating and grows as you speak. Streaming a test sentence through this exact pipeline produced this sequence of messages on the wire:\n\n```\n{\"type\": \"partial\", \"text\": \"Hello World!\"}\n{\"type\": \"partial\", \"text\": \"Hello World, this is a live transcription test streaming\"}\n{\"type\": \"partial\", \"text\": \"Hello World, this is a live transcription test streaming audio over a web socket.\"}\n```\n\nNotice the second partial extends and corrects the first. That's the re-transcription window doing its job.\n\n## Troubleshooting\n\n- \n**`TypeError: handler() missing 1 required positional argument: 'path'`** — you copied a handler signature from an old tutorial. The legacy API called`handler(websocket, path)` ; the current`websockets.asyncio` API passes only the connection. Delete the`path` parameter.\n- \n**`NotSupportedError: Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported.`** — Firefox. It won't resample a mic stream into a 16 kHz context (Mozilla bug 1725336). Use Chrome, Edge, or Safari, or create the context at the default rate and downsample inside the worklet before converting to int16.\n- \n**`Unable to load any of {libcudnn_ops.so.9.1.0, libcudnn_ops.so.9.1, libcudnn_ops.so.9, libcudnn_ops.so}`** — you switched to` device=\"cuda\"` without cuDNN 9. Run`pip install nvidia-cublas-cu12 nvidia-cudnn-cu12` and add both packages'`lib` directories to`LD_LIBRARY_PATH` , or stay on CPU with`compute_type=\"int8\"` .\n- \n**`OSError: [Errno 48] error while attempting to bind on address ('127.0.0.1', 8765)`** — a previous server instance is still holding the port (errno 98 on Linux). Find it with`lsof -nP -iTCP:8765` and kill it.\n\n## Next steps\n\nThe 15-second commit window is the crudest part of this design: text can flicker until it's committed. Production streaming systems solve that with the LocalAgreement policy, where a prefix is committed once two consecutive transcriptions agree on it. [whisper_streaming](https://github.com/ufal/whisper_streaming) implements it on top of faster-whisper and is a natural next read. On the model side, swap `base` for `distil-large-v3` or `large-v3` with `device=\"cuda\", compute_type=\"float16\"` for much better accuracy at similar latency. And before exposing this beyond localhost, put the socket behind TLS (`wss://`), because browsers block mixed-content WebSockets and mic capture on insecure origins anyway.\n\n## Sources & further reading\n\n1. \n                                    [faster-whisper 1.2.1](https://pypi.org/project/faster-whisper/)\n                                — pypi.org\n2. \n                                    [SYSTRAN/faster-whisper](https://github.com/SYSTRAN/faster-whisper)\n                                — github.com\n3. \n                                    [websockets 17.1 documentation](https://websockets.readthedocs.io/en/stable/)\n                                — websockets.readthedocs.io\n4. \n                                    [AudioContext() constructor](https://developer.mozilla.org/en-US/docs/Web/API/AudioContext/AudioContext)\n                                — developer.mozilla.org\n5. \n                                    [Error in AudioContext.createMediaStreamSource with custom sample rate](https://bugzilla.mozilla.org/show_bug.cgi?id=1725336)\n                                — bugzilla.mozilla.org\n\n[Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer\n\nPriya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.\n\n## Discussion 1\n\nthe firefox limitation is frustrating but honestly self-hosting whisper kills any native solution anyway. you're stuck doing your own audio resampling in js either way if you want cross-browser support — the article just sidesteps it entirely rather than showing how to actually handle it. would've been more useful than just listing it as a known issue.", "url": "https://wpnews.pro/news/build-a-real-time-speech-to-text-pipeline-with-whisper-and-websockets", "canonical_source": "https://sourcefeed.dev/a/build-a-real-time-speech-to-text-pipeline-with-whisper-and-websockets", "published_at": "2026-09-13 17:45:18+00:00", "updated_at": "2026-09-13 18:21:26.613662+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "natural-language-processing", "ai-products"], "entities": ["Priya Nair", "Whisper", "faster-whisper", "websockets", "CTranslate2", "OpenAI", "Hugging Face", "Python"], "alternates": {"html": "https://wpnews.pro/news/build-a-real-time-speech-to-text-pipeline-with-whisper-and-websockets", "markdown": "https://wpnews.pro/news/build-a-real-time-speech-to-text-pipeline-with-whisper-and-websockets.md", "text": "https://wpnews.pro/news/build-a-real-time-speech-to-text-pipeline-with-whisper-and-websockets.txt", "jsonld": "https://wpnews.pro/news/build-a-real-time-speech-to-text-pipeline-with-whisper-and-websockets.jsonld"}}