{"slug": "this-is-how-i-added-an-in-browser-auto-captions-feature-to-my-youtube-shorts-web", "title": "This is how I added an in-browser auto captions feature to my YouTube Shorts converter web application using Whisper AI and ffmpeg.wasm", "summary": "A developer added an in-browser auto captions feature to Convert to Shorts, a browser-based tool that converts horizontal videos to YouTube Shorts format, using Whisper AI and ffmpeg.wasm. The feature runs entirely client-side to preserve privacy, extracting audio via the Web Audio API, transcribing with Transformers.js, and burning captions using ASS subtitles after drawtext filters proved unreliable.", "body_md": "A few weeks ago I launched Convert to Shorts — a free browser-based tool that converts horizontal videos to YouTube Shorts format (9:16) without uploading anything to a server. I wrote about the ffmpeg.wasm + Vite setup in a previous article.\n\nThe most requested feature after launch was auto captions. Captions significantly boost Shorts engagement since most people watch without sound, and manually typing captions is tedious.\n\nThe challenge: how do you add free auto captions to a privacy-first tool that never uploads your video to a server?\n\nThe answer: run Whisper AI in the browser.\n\nThe stack -\n\n`@xenova/transformers`) — Hugging Face's JavaScript port of the Transformers library, runs ONNX models in the browser via WebAssembly\n**Step 1: Audio extraction**\n\nWhisper expects mono 16kHz audio as a Float32Array. The Web Audio API handles this cleanly:\n\n```\nasync function extractAudio(\n  file: File,\n  trimStart: number,\n  trimEnd: number\n): Promise<Float32Array> {\n  const arrayBuffer = await file.arrayBuffer();\n  const audioContext = new AudioContext({ sampleRate: 16000 });\n  const audioBuffer = await audioContext.decodeAudioData(arrayBuffer);\n\n  const sampleRate = audioContext.sampleRate;\n  const startSample = Math.floor(trimStart * sampleRate);\n  const endSample = Math.floor(trimEnd * sampleRate);\n\n  // Mix down to mono, slice to trim range\n  const channelData = audioBuffer.getChannelData(0);\n  const trimmed = channelData.slice(startSample, endSample);\n  await audioContext.close();\n  return trimmed;\n}\n```\n\nCreating the `AudioContext` at 16kHz means the browser automatically resamples from whatever the source rate is (usually 44.1kHz or 48kHz). No manual resampling needed.\n\n**Step 2: Running Whisper**\n\nTransformers.js makes this surprisingly straightforward:\n\n```\nconst { pipeline, env } = await import(\"@xenova/transformers\");\n\nenv.useBrowserCache = true;\n\nconst transcriber = await pipeline(\n  \"automatic-speech-recognition\",\n  \"Xenova/whisper-tiny\",\n  {\n    progress_callback: (p) => {\n      if (p.status === \"downloading\") {\n        setModelProgress(Math.round((p.loaded / p.total) * 100));\n      }\n    },\n  }\n);\n\nconst result = await transcriber(audioFloat32Array, {\n  return_timestamps: true,\n  chunk_length_s: 30,\n  stride_length_s: 5,\n});\n```\n\n`return_timestamps: true` gives you back an array of segments with `text` and `timestamp: [start, end]` — exactly what you need for timed captions.\n\n`env.useBrowserCache = true` means the model downloads once and is stored in IndexedDB. Every subsequent use loads from cache — no 75MB download each time.\n\n**Step 3: Burning captions with ffmpeg.wasm**\n\nThis is where it got interesting. My first instinct was to use ffmpeg's `drawtext` filter with an `enable='between(t,start,end)'` expression for each caption segment — one filter per caption, chained with commas.\n\nThis failed in multiple ways:\n\n`between(t,start,end)` was interpreted as a filter separator`\\\\`, didn't work in ffmpeg.wasm's argument parsing` gte(t,start)*lte(t,end)` instead of between also failed at the filter chain level`drawtext` filters caused ffmpeg.wasm to abort\nThe reliable solution was ASS subtitles. ASS (Advanced SubStation Alpha) is a subtitle format that ffmpeg's built-in `ass` filter handles natively via libass:\n\n```\nfunction buildAssSubtitles(segments, color, size) {\n  function toAssTime(seconds) {\n    const h = Math.floor(seconds / 3600);\n    const m = Math.floor((seconds % 3600) / 60);\n    const s = Math.floor(seconds % 60);\n    const cs = Math.round((seconds % 1) * 100);\n    return `${h}:${String(m).padStart(2,\"0\")}:${String(s).padStart(2,\"0\")}.${String(cs).padStart(2,\"0\")}`;\n  }\n\n  const header = `[Script Info]\nScriptType: v4.00+\nPlayResX: 1080\nPlayResY: 1920\nWrapStyle: 1\nScaledBorderAndShadow: yes\n\n[V4+ Styles]\nFormat: Name, Fontname, Fontsize, PrimaryColour, ...\nStyle: Default,Roboto Bold,75,&H00FFFFFF,...\n\n[Events]\nFormat: Layer, Start, End, Style, Text\n`;\n\n  const events = segments\n    .map(seg =>\n      `Dialogue: 0,${toAssTime(seg.start)},${toAssTime(seg.end)},Default,${seg.text}`\n    )\n    .join(\"\\n\");\n\n  return header + events;\n}\n```\n\n**Step 4: The font problem**\n\nlibass needs a font to render subtitles. In a normal environment it uses system fonts. In ffmpeg.wasm's WebAssembly sandbox there are no system fonts.\n\nThree approaches I tried:\n\nApproach 1: Embed font as Base64 in the ASS file\n\nThe ASS format supports a `[Fonts]` section with Base64-encoded font data split into 80-character lines. This failed with a libass assertion error in `ass.c` about Base64 padding — the ffmpeg.wasm build of libass appears to have a bug in its font decoder.\n\nApproach 2: Write font to ffmpeg.wasm virtual filesystem and use `fontsdir`\n\n```\nass=captions.ass:fontsdir=/fonts\n```\n\nWrite the font file to `/fonts/Roboto-Bold.ttf` in the virtual filesystem before running ffmpeg. This worked.\n\n``` js\nawait ffmpeg.createDir(\"/fonts\");\nconst fontResponse = await fetch(\"/Roboto-Bold.ttf\");\nconst fontBuffer = await fontResponse.arrayBuffer();\nawait ffmpeg.writeFile(\"/fonts/Roboto-Bold.ttf\", new Uint8Array(fontBuffer));\n```\n\nThe key insight: ffmpeg.wasm has a full virtual filesystem (Emscripten's FS). You can create directories and write files to it just like a real filesystem, and ffmpeg commands can reference those paths.\n\nAs a result the full caption pipeline added roughly 10-30 seconds to the export time for a 30-60 second clip. The Whisper tiny model is surprisingly accurate for clear English speech. Multiple languages work out of the box since Whisper was trained on multilingual data.\n\n**What I learned**\n\nTransformers.js is genuinely production-ready. The API is clean, browser caching just works, and the ONNX runtime handles the WebAssembly execution reliably.\n\nASS subtitles are more robust than `drawtext` filter chains. If you're burning timed text into video with ffmpeg.wasm, reach for the `ass` filter before trying to chain multiple drawtext filters.\n\nffmpeg.wasm's virtual filesystem is powerful. You're not limited to just reading and writing video files — you can create directory structures, write fonts, write subtitle files, and reference them all from ffmpeg commands exactly as you would on a real filesystem.\n\nlibass has quirks in the WASM build. The embedded font Base64 decoding in the WASM build of libass appears broken. Use fontsdir instead.\n\nTry it\n\n[converttoshorts.com](https://converttoshorts.com/) — free, no account, no upload. Auto captions are in the export panel.", "url": "https://wpnews.pro/news/this-is-how-i-added-an-in-browser-auto-captions-feature-to-my-youtube-shorts-web", "canonical_source": "https://dev.to/dhritich20baruah/this-is-how-i-added-an-in-browser-auto-captions-feature-to-my-youtube-shorts-converter-web-41o0", "published_at": "2026-09-07 03:23:29+00:00", "updated_at": "2026-09-07 03:28:34.584202+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "machine-learning", "ai-tools"], "entities": ["Convert to Shorts", "Whisper AI", "ffmpeg.wasm", "Transformers.js", "Hugging Face", "Xenova", "YouTube"], "alternates": {"html": "https://wpnews.pro/news/this-is-how-i-added-an-in-browser-auto-captions-feature-to-my-youtube-shorts-web", "markdown": "https://wpnews.pro/news/this-is-how-i-added-an-in-browser-auto-captions-feature-to-my-youtube-shorts-web.md", "text": "https://wpnews.pro/news/this-is-how-i-added-an-in-browser-auto-captions-feature-to-my-youtube-shorts-web.txt", "jsonld": "https://wpnews.pro/news/this-is-how-i-added-an-in-browser-auto-captions-feature-to-my-youtube-shorts-web.jsonld"}}