# This is how I added an in-browser auto captions feature to my YouTube Shorts converter web application using Whisper AI and ffmpeg.wasm

> Source: <https://dev.to/dhritich20baruah/this-is-how-i-added-an-in-browser-auto-captions-feature-to-my-youtube-shorts-converter-web-41o0>
> Published: 2026-09-07 03:23:29+00:00

A few weeks ago I launched Convert to Shorts — a free browser-based tool that converts horizontal videos to YouTube Shorts format (9:16) without uploading anything to a server. I wrote about the ffmpeg.wasm + Vite setup in a previous article.

The most requested feature after launch was auto captions. Captions significantly boost Shorts engagement since most people watch without sound, and manually typing captions is tedious.

The challenge: how do you add free auto captions to a privacy-first tool that never uploads your video to a server?

The answer: run Whisper AI in the browser.

The stack -

`@xenova/transformers`) — Hugging Face's JavaScript port of the Transformers library, runs ONNX models in the browser via WebAssembly
**Step 1: Audio extraction**

Whisper expects mono 16kHz audio as a Float32Array. The Web Audio API handles this cleanly:

```
async function extractAudio(
  file: File,
  trimStart: number,
  trimEnd: number
): Promise<Float32Array> {
  const arrayBuffer = await file.arrayBuffer();
  const audioContext = new AudioContext({ sampleRate: 16000 });
  const audioBuffer = await audioContext.decodeAudioData(arrayBuffer);

  const sampleRate = audioContext.sampleRate;
  const startSample = Math.floor(trimStart * sampleRate);
  const endSample = Math.floor(trimEnd * sampleRate);

  // Mix down to mono, slice to trim range
  const channelData = audioBuffer.getChannelData(0);
  const trimmed = channelData.slice(startSample, endSample);
  await audioContext.close();
  return trimmed;
}
```

Creating the `AudioContext` at 16kHz means the browser automatically resamples from whatever the source rate is (usually 44.1kHz or 48kHz). No manual resampling needed.

**Step 2: Running Whisper**

Transformers.js makes this surprisingly straightforward:

```
const { pipeline, env } = await import("@xenova/transformers");

env.useBrowserCache = true;

const transcriber = await pipeline(
  "automatic-speech-recognition",
  "Xenova/whisper-tiny",
  {
    progress_callback: (p) => {
      if (p.status === "downloading") {
        setModelProgress(Math.round((p.loaded / p.total) * 100));
      }
    },
  }
);

const result = await transcriber(audioFloat32Array, {
  return_timestamps: true,
  chunk_length_s: 30,
  stride_length_s: 5,
});
```

`return_timestamps: true` gives you back an array of segments with `text` and `timestamp: [start, end]` — exactly what you need for timed captions.

`env.useBrowserCache = true` means the model downloads once and is stored in IndexedDB. Every subsequent use loads from cache — no 75MB download each time.

**Step 3: Burning captions with ffmpeg.wasm**

This is where it got interesting. My first instinct was to use ffmpeg's `drawtext` filter with an `enable='between(t,start,end)'` expression for each caption segment — one filter per caption, chained with commas.

This failed in multiple ways:

`between(t,start,end)` was interpreted as a filter separator`\\`, didn't work in ffmpeg.wasm's argument parsing` gte(t,start)*lte(t,end)` instead of between also failed at the filter chain level`drawtext` filters caused ffmpeg.wasm to abort
The reliable solution was ASS subtitles. ASS (Advanced SubStation Alpha) is a subtitle format that ffmpeg's built-in `ass` filter handles natively via libass:

```
function buildAssSubtitles(segments, color, size) {
  function toAssTime(seconds) {
    const h = Math.floor(seconds / 3600);
    const m = Math.floor((seconds % 3600) / 60);
    const s = Math.floor(seconds % 60);
    const cs = Math.round((seconds % 1) * 100);
    return `${h}:${String(m).padStart(2,"0")}:${String(s).padStart(2,"0")}.${String(cs).padStart(2,"0")}`;
  }

  const header = `[Script Info]
ScriptType: v4.00+
PlayResX: 1080
PlayResY: 1920
WrapStyle: 1
ScaledBorderAndShadow: yes

[V4+ Styles]
Format: Name, Fontname, Fontsize, PrimaryColour, ...
Style: Default,Roboto Bold,75,&H00FFFFFF,...

[Events]
Format: Layer, Start, End, Style, Text
`;

  const events = segments
    .map(seg =>
      `Dialogue: 0,${toAssTime(seg.start)},${toAssTime(seg.end)},Default,${seg.text}`
    )
    .join("\n");

  return header + events;
}
```

**Step 4: The font problem**

libass needs a font to render subtitles. In a normal environment it uses system fonts. In ffmpeg.wasm's WebAssembly sandbox there are no system fonts.

Three approaches I tried:

Approach 1: Embed font as Base64 in the ASS file

The ASS format supports a `[Fonts]` section with Base64-encoded font data split into 80-character lines. This failed with a libass assertion error in `ass.c` about Base64 padding — the ffmpeg.wasm build of libass appears to have a bug in its font decoder.

Approach 2: Write font to ffmpeg.wasm virtual filesystem and use `fontsdir`

```
ass=captions.ass:fontsdir=/fonts
```

Write the font file to `/fonts/Roboto-Bold.ttf` in the virtual filesystem before running ffmpeg. This worked.

``` js
await ffmpeg.createDir("/fonts");
const fontResponse = await fetch("/Roboto-Bold.ttf");
const fontBuffer = await fontResponse.arrayBuffer();
await ffmpeg.writeFile("/fonts/Roboto-Bold.ttf", new Uint8Array(fontBuffer));
```

The key insight: ffmpeg.wasm has a full virtual filesystem (Emscripten's FS). You can create directories and write files to it just like a real filesystem, and ffmpeg commands can reference those paths.

As a result the full caption pipeline added roughly 10-30 seconds to the export time for a 30-60 second clip. The Whisper tiny model is surprisingly accurate for clear English speech. Multiple languages work out of the box since Whisper was trained on multilingual data.

**What I learned**

Transformers.js is genuinely production-ready. The API is clean, browser caching just works, and the ONNX runtime handles the WebAssembly execution reliably.

ASS subtitles are more robust than `drawtext` filter chains. If you're burning timed text into video with ffmpeg.wasm, reach for the `ass` filter before trying to chain multiple drawtext filters.

ffmpeg.wasm's virtual filesystem is powerful. You're not limited to just reading and writing video files — you can create directory structures, write fonts, write subtitle files, and reference them all from ffmpeg commands exactly as you would on a real filesystem.

libass has quirks in the WASM build. The embedded font Base64 decoding in the WASM build of libass appears broken. Use fontsdir instead.

Try it

[converttoshorts.com](https://converttoshorts.com/) — free, no account, no upload. Auto captions are in the export panel.