This is how I added an in-browser auto captions feature to my YouTube Shorts converter web application using Whisper AI and ffmpeg.wasm

A developer added an in-browser auto captions feature to Convert to Shorts, a browser-based tool that converts horizontal videos to YouTube Shorts format, using Whisper AI and ffmpeg.wasm. The feature runs entirely client-side to preserve privacy, extracting audio via the Web Audio API, transcribing with Transformers.js, and burning captions using ASS subtitles after drawtext filters proved unreliable.

A few weeks ago I launched Convert to Shorts — a free browser-based tool that converts horizontal videos to YouTube Shorts format 9:16 without uploading anything to a server. I wrote about the ffmpeg.wasm + Vite setup in a previous article. The most requested feature after launch was auto captions. Captions significantly boost Shorts engagement since most people watch without sound, and manually typing captions is tedious. The challenge: how do you add free auto captions to a privacy-first tool that never uploads your video to a server? The answer: run Whisper AI in the browser. The stack - @xenova/transformers — Hugging Face's JavaScript port of the Transformers library, runs ONNX models in the browser via WebAssembly Step 1: Audio extraction Whisper expects mono 16kHz audio as a Float32Array. The Web Audio API handles this cleanly: async function extractAudio file: File, trimStart: number, trimEnd: number : Promise<Float32Array { const arrayBuffer = await file.arrayBuffer ; const audioContext = new AudioContext { sampleRate: 16000 } ; const audioBuffer = await audioContext.decodeAudioData arrayBuffer ; const sampleRate = audioContext.sampleRate; const startSample = Math.floor trimStart sampleRate ; const endSample = Math.floor trimEnd sampleRate ; // Mix down to mono, slice to trim range const channelData = audioBuffer.getChannelData 0 ; const trimmed = channelData.slice startSample, endSample ; await audioContext.close ; return trimmed; } Creating the AudioContext at 16kHz means the browser automatically resamples from whatever the source rate is usually 44.1kHz or 48kHz . No manual resampling needed. Step 2: Running Whisper Transformers.js makes this surprisingly straightforward: const { pipeline, env } = await import "@xenova/transformers" ; env.useBrowserCache = true; const transcriber = await pipeline "automatic-speech-recognition", "Xenova/whisper-tiny", { progress callback: p = { if p.status === "downloading" { setModelProgress Math.round p.loaded / p.total 100 ; } }, } ; const result = await transcriber audioFloat32Array, { return timestamps: true, chunk length s: 30, stride length s: 5, } ; return timestamps: true gives you back an array of segments with text and timestamp: start, end — exactly what you need for timed captions. env.useBrowserCache = true means the model downloads once and is stored in IndexedDB. Every subsequent use loads from cache — no 75MB download each time. Step 3: Burning captions with ffmpeg.wasm This is where it got interesting. My first instinct was to use ffmpeg's drawtext filter with an enable='between t,start,end ' expression for each caption segment — one filter per caption, chained with commas. This failed in multiple ways: between t,start,end was interpreted as a filter separator \\ , didn't work in ffmpeg.wasm's argument parsing gte t,start lte t,end instead of between also failed at the filter chain level drawtext filters caused ffmpeg.wasm to abort The reliable solution was ASS subtitles. ASS Advanced SubStation Alpha is a subtitle format that ffmpeg's built-in ass filter handles natively via libass: function buildAssSubtitles segments, color, size { function toAssTime seconds { const h = Math.floor seconds / 3600 ; const m = Math.floor seconds % 3600 / 60 ; const s = Math.floor seconds % 60 ; const cs = Math.round seconds % 1 100 ; return ${h}:${String m .padStart 2,"0" }:${String s .padStart 2,"0" }.${String cs .padStart 2,"0" } ; } const header = Script Info ScriptType: v4.00+ PlayResX: 1080 PlayResY: 1920 WrapStyle: 1 ScaledBorderAndShadow: yes V4+ Styles Format: Name, Fontname, Fontsize, PrimaryColour, ... Style: Default,Roboto Bold,75,&H00FFFFFF,... Events Format: Layer, Start, End, Style, Text ; const events = segments .map seg = Dialogue: 0,${toAssTime seg.start },${toAssTime seg.end },Default,${seg.text} .join "\n" ; return header + events; } Step 4: The font problem libass needs a font to render subtitles. In a normal environment it uses system fonts. In ffmpeg.wasm's WebAssembly sandbox there are no system fonts. Three approaches I tried: Approach 1: Embed font as Base64 in the ASS file The ASS format supports a Fonts section with Base64-encoded font data split into 80-character lines. This failed with a libass assertion error in ass.c about Base64 padding — the ffmpeg.wasm build of libass appears to have a bug in its font decoder. Approach 2: Write font to ffmpeg.wasm virtual filesystem and use fontsdir ass=captions.ass:fontsdir=/fonts Write the font file to /fonts/Roboto-Bold.ttf in the virtual filesystem before running ffmpeg. This worked. js await ffmpeg.createDir "/fonts" ; const fontResponse = await fetch "/Roboto-Bold.ttf" ; const fontBuffer = await fontResponse.arrayBuffer ; await ffmpeg.writeFile "/fonts/Roboto-Bold.ttf", new Uint8Array fontBuffer ; The key insight: ffmpeg.wasm has a full virtual filesystem Emscripten's FS . You can create directories and write files to it just like a real filesystem, and ffmpeg commands can reference those paths. As a result the full caption pipeline added roughly 10-30 seconds to the export time for a 30-60 second clip. The Whisper tiny model is surprisingly accurate for clear English speech. Multiple languages work out of the box since Whisper was trained on multilingual data. What I learned Transformers.js is genuinely production-ready. The API is clean, browser caching just works, and the ONNX runtime handles the WebAssembly execution reliably. ASS subtitles are more robust than drawtext filter chains. If you're burning timed text into video with ffmpeg.wasm, reach for the ass filter before trying to chain multiple drawtext filters. ffmpeg.wasm's virtual filesystem is powerful. You're not limited to just reading and writing video files — you can create directory structures, write fonts, write subtitle files, and reference them all from ffmpeg commands exactly as you would on a real filesystem. libass has quirks in the WASM build. The embedded font Base64 decoding in the WASM build of libass appears broken. Use fontsdir instead. Try it converttoshorts.com https://converttoshorts.com/ — free, no account, no upload. Auto captions are in the export panel.