A few weeks ago I launched Convert to Shorts — a free browser-based tool that converts horizontal videos to YouTube Shorts format (9:16) without up anything to a server. I wrote about the ffmpeg.wasm + Vite setup in a previous article.
The most requested feature after launch was auto captions. Captions significantly boost Shorts engagement since most people watch without sound, and manually typing captions is tedious.
The challenge: how do you add free auto captions to a privacy-first tool that never uploads your video to a server?
The answer: run Whisper AI in the browser.
The stack -
@xenova/transformers) — Hugging Face's JavaScript port of the Transformers library, runs ONNX models in the browser via WebAssembly
Step 1: Audio extraction
Whisper expects mono 16kHz audio as a Float32Array. The Web Audio API handles this cleanly:
async function extractAudio(
file: File,
trimStart: number,
trimEnd: number
): Promise<Float32Array> {
const arrayBuffer = await file.arrayBuffer();
const audioContext = new AudioContext({ sampleRate: 16000 });
const audioBuffer = await audioContext.decodeAudioData(arrayBuffer);
const sampleRate = audioContext.sampleRate;
const startSample = Math.floor(trimStart * sampleRate);
const endSample = Math.floor(trimEnd * sampleRate);
// Mix down to mono, slice to trim range
const channelData = audioBuffer.getChannelData(0);
const trimmed = channelData.slice(startSample, endSample);
await audioContext.close();
return trimmed;
}
Creating the AudioContext at 16kHz means the browser automatically resamples from whatever the source rate is (usually 44.1kHz or 48kHz). No manual resampling needed.
Step 2: Running Whisper
Transformers.js makes this surprisingly straightforward:
const { pipeline, env } = await import("@xenova/transformers");
env.useBrowserCache = true;
const transcriber = await pipeline(
"automatic-speech-recognition",
"Xenova/whisper-tiny",
{
progress_callback: (p) => {
if (p.status === "down") {
setModelProgress(Math.round((p.loaded / p.total) * 100));
}
},
}
);
const result = await transcriber(audioFloat32Array, {
return_timestamps: true,
chunk_length_s: 30,
stride_length_s: 5,
});
return_timestamps: true gives you back an array of segments with text and timestamp: [start, end] — exactly what you need for timed captions.
env.useBrowserCache = true means the model downloads once and is stored in IndexedDB. Every subsequent use loads from cache — no 75MB download each time.
Step 3: Burning captions with ffmpeg.wasm
This is where it got interesting. My first instinct was to use ffmpeg's drawtext filter with an enable='between(t,start,end)' expression for each caption segment — one filter per caption, chained with commas.
This failed in multiple ways:
between(t,start,end) was interpreted as a filter separator\\, didn't work in ffmpeg.wasm's argument parsing gte(t,start)*lte(t,end) instead of between also failed at the filter chain leveldrawtext filters caused ffmpeg.wasm to abort
The reliable solution was ASS subtitles. ASS (Advanced SubStation Alpha) is a subtitle format that ffmpeg's built-in ass filter handles natively via libass:
function buildAssSubtitles(segments, color, size) {
function toAssTime(seconds) {
const h = Math.floor(seconds / 3600);
const m = Math.floor((seconds % 3600) / 60);
const s = Math.floor(seconds % 60);
const cs = Math.round((seconds % 1) * 100);
return `${h}:${String(m).padStart(2,"0")}:${String(s).padStart(2,"0")}.${String(cs).padStart(2,"0")}`;
}
const header = `[Script Info]
ScriptType: v4.00+
PlayResX: 1080
PlayResY: 1920
WrapStyle: 1
ScaledBorderAndShadow: yes
[V4+ Styles]
Format: Name, Fontname, Fontsize, PrimaryColour, ...
Style: Default,Roboto Bold,75,&H00FFFFFF,...
[Events]
Format: Layer, Start, End, Style, Text
`;
const events = segments
.map(seg =>
`Dialogue: 0,${toAssTime(seg.start)},${toAssTime(seg.end)},Default,${seg.text}`
)
.join("\n");
return header + events;
}
Step 4: The font problem
libass needs a font to render subtitles. In a normal environment it uses system fonts. In ffmpeg.wasm's WebAssembly sandbox there are no system fonts.
Three approaches I tried:
Approach 1: Embed font as Base64 in the ASS file
The ASS format supports a [Fonts] section with Base64-encoded font data split into 80-character lines. This failed with a libass assertion error in ass.c about Base64 padding — the ffmpeg.wasm build of libass appears to have a bug in its font decoder.
Approach 2: Write font to ffmpeg.wasm virtual filesystem and use fontsdir
ass=captions.ass:fontsdir=/fonts
Write the font file to /fonts/Roboto-Bold.ttf in the virtual filesystem before running ffmpeg. This worked.
await ffmpeg.createDir("/fonts");
const fontResponse = await fetch("/Roboto-Bold.ttf");
const fontBuffer = await fontResponse.arrayBuffer();
await ffmpeg.writeFile("/fonts/Roboto-Bold.ttf", new Uint8Array(fontBuffer));
The key insight: ffmpeg.wasm has a full virtual filesystem (Emscripten's FS). You can create directories and write files to it just like a real filesystem, and ffmpeg commands can reference those paths.
As a result the full caption pipeline added roughly 10-30 seconds to the export time for a 30-60 second clip. The Whisper tiny model is surprisingly accurate for clear English speech. Multiple languages work out of the box since Whisper was trained on multilingual data.
What I learned
Transformers.js is genuinely production-ready. The API is clean, browser caching just works, and the ONNX runtime handles the WebAssembly execution reliably.
ASS subtitles are more robust than drawtext filter chains. If you're burning timed text into video with ffmpeg.wasm, reach for the ass filter before trying to chain multiple drawtext filters.
ffmpeg.wasm's virtual filesystem is powerful. You're not limited to just reading and writing video files — you can create directory structures, write fonts, write subtitle files, and reference them all from ffmpeg commands exactly as you would on a real filesystem.
libass has quirks in the WASM build. The embedded font Base64 decoding in the WASM build of libass appears broken. Use fontsdir instead.
Try it
converttoshorts.com — free, no account, no upload. Auto captions are in the export panel.