Stream microphone audio to a self-hosted Whisper server over WebSockets and render live captions in the browser.
What you'll build #
A browser page that streams your microphone to a self-hosted Whisper model over a WebSocket and shows live captions: gray interim text that updates about once a second, committed black text every 15 seconds. Everything runs on your machine. No audio leaves it.
Prerequisites #
Verified against faster-whisper 1.2.1, websockets 17.1, and ctranslate2 4.8.2 on Python 3.13 (macOS 15). Linux and Windows work the same way.
- Python 3.11 or newer (websockets 17.x requires 3.11+)
- Chrome, Edge, or Safari. Firefox can't resample mic input to a 16 kHz AudioContext; see Troubleshooting
- ~150 MB of disk for the
basemodel, downloaded from Hugging Face on first run - A CPU is enough for the
basemodel. GPU needs CUDA 12 and cuDNN 9
Step 1: Install the server dependencies #
faster-whisper runs Whisper on CTranslate2, about 4x faster than the reference implementation, which is what makes once-a-second re-transcription workable on a laptop CPU. websockets handles the socket side.
mkdir whisper-live && cd whisper-live
python3 -m venv venv && source venv/bin/activate
pip install "faster-whisper==1.2.1" "websockets==17.1"
Step 2: Write the transcription server #
The protocol is deliberately dumb: the client sends raw 16-bit little-endian PCM at 16 kHz as binary frames, the server replies with JSON. The server appends every frame to a buffer and, once a second, re-transcribes the whole buffer and pushes the result as a partial. When the buffer hits 15 seconds it commits the text as a final and starts fresh. Re-transcribing the growing buffer costs more than a true incremental decoder, but it self-corrects earlier words as context arrives and needs no alignment logic.
Save this as server.py:
import asyncio
import json
import numpy as np
from faster_whisper import WhisperModel
from websockets.asyncio.server import serve
SAMPLE_RATE = 16000
BYTES_PER_SECOND = SAMPLE_RATE * 2 # 16-bit mono PCM
MAX_WINDOW_SECONDS = 15
model = WhisperModel("base", device="cpu", compute_type="int8")
def transcribe(pcm: bytes) -> str:
audio = np.frombuffer(pcm, dtype=np.int16).astype(np.float32) / 32768.0
segments, _ = model.transcribe(
audio,
language="en",
beam_size=1,
vad_filter=True,
condition_on_previous_text=False,
)
return " ".join(segment.text.strip() for segment in segments)
async def handler(websocket):
buffer = bytearray()
async def worker():
last_len = 0
while True:
await asyncio.sleep(1.0)
if len(buffer) == last_len or len(buffer) < BYTES_PER_SECOND:
continue
snapshot = bytes(buffer)
last_len = len(snapshot)
text = await asyncio.to_thread(transcribe, snapshot)
if len(snapshot) >= MAX_WINDOW_SECONDS * BYTES_PER_SECOND:
del buffer[: len(snapshot)]
last_len = 0
await websocket.send(json.dumps({"type": "final", "text": text}))
else:
await websocket.send(json.dumps({"type": "partial", "text": text}))
task = asyncio.create_task(worker())
try:
async for message in websocket:
if isinstance(message, bytes):
buffer.extend(message)
finally:
task.cancel()
async def main():
async with serve(handler, "localhost", 8765) as server:
print("Listening on ws://localhost:8765")
await server.serve_forever()
if __name__ == "__main__":
asyncio.run(main())
Four choices matter here. model.transcribe accepts a float32 numpy array at 16 kHz directly, so there's no temp-file dance. asyncio.to_thread keeps the CPU-bound decode off the event loop, so incoming frames keep landing in the buffer while the model runs. beam_size=1 with condition_on_previous_text=False trades a little accuracy for latency and stops the model hallucinating repeats on partial audio. vad_filter=True runs Silero VAD first, so silence and keyboard noise don't get decoded into phantom words.
Step 3: Build the browser client #
Creating the AudioContext at 16 kHz makes the browser resample the mic for you, so the worklet only has to convert float32 samples to int16. Save this as index.html:
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Live transcription</title>
</head>
<body>
<button id="start">Start listening</button>
<p id="status">Idle</p>
<p id="committed"></p>
<p id="partial" style="color: #888"></p>
<script>
const status = document.getElementById("status");
const committed = document.getElementById("committed");
const partial = document.getElementById("partial");
document.getElementById("start").onclick = async () => {
const ws = new WebSocket("ws://localhost:8765");
ws.onmessage = (event) => {
const msg = JSON.parse(event.data);
if (msg.type === "final") {
committed.textContent += msg.text + " ";
partial.textContent = "";
} else {
partial.textContent = msg.text;
}
};
await new Promise((resolve) => (ws.onopen = resolve));
const stream = await navigator.mediaDevices.getUserMedia({
audio: { channelCount: 1, echoCancellation: true, noiseSuppression: true },
});
// 16 kHz context: the browser resamples the mic to Whisper's input rate.
const context = new AudioContext({ sampleRate: 16000 });
await context.audioWorklet.addModule("pcm-processor.js");
const source = context.createMediaStreamSource(stream);
const node = new AudioWorkletNode(context, "pcm-processor");
node.port.onmessage = (event) => {
if (ws.readyState === WebSocket.OPEN) ws.send(event.data);
};
source.connect(node);
node.connect(context.destination); // keeps the graph pulled; the node outputs silence
status.textContent = "Listening...";
};
</script>
</body>
</html>
Step 4: Add the AudioWorklet processor #
An AudioWorklet hands you audio in 128-frame chunks on the audio thread. Sending a WebSocket message every 8 ms would be wasteful, so the processor batches 4096 samples (~256 ms) per message. Save this as pcm-processor.js next to index.html:
class PCMProcessor extends AudioWorkletProcessor {
constructor() {
super();
this.samples = [];
this.chunkSize = 4096; // ~256 ms at 16 kHz per WebSocket message
}
process(inputs) {
const input = inputs[0][0];
if (!input) return true;
for (let i = 0; i < input.length; i++) this.samples.push(input[i]);
if (this.samples.length >= this.chunkSize) {
const pcm = new Int16Array(this.samples.length);
for (let i = 0; i < this.samples.length; i++) {
const s = Math.max(-1, Math.min(1, this.samples[i]));
pcm[i] = s < 0 ? s * 0x8000 : s * 0x7fff;
}
this.port.postMessage(pcm.buffer, [pcm.buffer]);
this.samples = [];
}
return true;
}
}
registerProcessor("pcm-processor", PCMProcessor);
Verify it works #
Start the server. The first run downloads the model, then you should see:
Listening on ws://localhost:8765
Serve the client from a second terminal in the same directory. getUserMedia needs a secure context and file:// doesn't qualify, but http://localhost does:
python3 -m http.server 8000
Open http://localhost:8000, click Start listening, allow the microphone, and talk. Within a couple of seconds the gray line starts updating and grows as you speak. Streaming a test sentence through this exact pipeline produced this sequence of messages on the wire:
{"type": "partial", "text": "Hello World!"}
{"type": "partial", "text": "Hello World, this is a live transcription test streaming"}
{"type": "partial", "text": "Hello World, this is a live transcription test streaming audio over a web socket."}
Notice the second partial extends and corrects the first. That's the re-transcription window doing its job.
Troubleshooting #
TypeError: handler() missing 1 required positional argument: 'path' — you copied a handler signature from an old tutorial. The legacy API calledhandler(websocket, path) ; the currentwebsockets.asyncio API passes only the connection. Delete thepath parameter. #
NotSupportedError: Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported. — Firefox. It won't resample a mic stream into a 16 kHz context (Mozilla bug 1725336). Use Chrome, Edge, or Safari, or create the context at the default rate and downsample inside the worklet before converting to int16. #
Unable to load any of {libcudnn_ops.so.9.1.0, libcudnn_ops.so.9.1, libcudnn_ops.so.9, libcudnn_ops.so} — you switched to device="cuda" without cuDNN 9. Runpip install nvidia-cublas-cu12 nvidia-cudnn-cu12 and add both packages'lib directories toLD_LIBRARY_PATH , or stay on CPU withcompute_type="int8" . #
OSError: [Errno 48] error while attempting to bind on address ('127.0.0.1', 8765) — a previous server instance is still holding the port (errno 98 on Linux). Find it withlsof -nP -iTCP:8765 and kill it.
Next steps #
The 15-second commit window is the crudest part of this design: text can flicker until it's committed. Production streaming systems solve that with the LocalAgreement policy, where a prefix is committed once two consecutive transcriptions agree on it. whisper_streaming implements it on top of faster-whisper and is a natural next read. On the model side, swap base for distil-large-v3 or large-v3 with device="cuda", compute_type="float16" for much better accuracy at similar latency. And before exposing this beyond localhost, put the socket behind TLS (wss://), because browsers block mixed-content WebSockets and mic capture on insecure origins anyway.
Sources & further reading #
-
[faster-whisper 1.2.1](https://pypi.org/project/faster-whisper/) — pypi.org -
[SYSTRAN/faster-whisper](https://github.com/SYSTRAN/faster-whisper) — github.com -
[websockets 17.1 documentation](https://websockets.readthedocs.io/en/stable/) — websockets.readthedocs.io -
[AudioContext() constructor](https://developer.mozilla.org/en-US/docs/Web/API/AudioContext/AudioContext) — developer.mozilla.org -
[Error in AudioContext.createMediaStreamSource with custom sample rate](https://bugzilla.mozilla.org/show_bug.cgi?id=1725336) — bugzilla.mozilla.org
Priya Nair· AI & Developer Experience Writer
Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.
Discussion 1 #
the firefox limitation is frustrating but honestly self-hosting whisper kills any native solution anyway. you're stuck doing your own audio resampling in js either way if you want cross-browser support — the article just sidesteps it entirely rather than showing how to actually handle it. would've been more useful than just listing it as a known issue.