Build an AI dubbing pipeline: faster-whisper + XTTS-v2 + FFmpeg

A developer built a script that takes an English video and produces a Spanish-narrated version with a cloned voice of the original speaker, using faster-whisper for transcription, an LLM for translation, XTTS-v2 for voice synthesis, and FFmpeg for audio replacement. The pipeline addresses the common problem of translated audio not fitting its time slot by enforcing length constraints per segment during translation.

TL;DR We're building a script that takes a video in English and produces the same video narrated in Spanish, in a cloned version of the original speaker's voice. Stack: faster-whisper for timestamped transcription, an LLM or any MT engine for translation, XTTS-v2 for voice-cloned synthesis, FFmpeg for surgery. We'll also handle the problem every demo skips: translated audio that doesn't fit its time slot. 📦 Code: github.com/USER/repo replace before publishing If you'd rather start from a finished system, Softcatala's open-dubbing https://github.com/softcatala/open-dubbing and KrillinAI https://github.com/krillinai/KrillinAI are full pipelines behind one CLI. This post builds the minimal version by hand so you understand what those tools are doing, and where they break. Python 3.10–3.12. The original Coqui company shut down in early 2024; the maintained fork of their TTS library is published by Idiap as coqui-tts : bash $ python -m venv dub && source dub/bin/activate $ pip install faster-whisper coqui-tts $ ffmpeg -version | head -1 6.0+ is fine, 8.x current ⚠️ Note: the XTTS-v2 model weightsship under the Coqui Public Model License, which restricts commercial use. Prototype freely, but before dubbed videos ship to paying customers, someone must read that license and possibly swap the synthesis step for a commercially licensed model or paid API. Voice cloning also requires the speaker's consent. Get it in writing. bash pull mono 16k audio for the ASR step $ ffmpeg -i input.mp4 -vn -ac 1 -ar 16000 -y source.wav python dub/transcribe.py from faster whisper import WhisperModel model = WhisperModel "large-v3-turbo", compute type="int8" segments, info = model.transcribe "source.wav", word timestamps=True lines = for seg in segments: lines.append { "start": seg.start, "end": seg.end, "text": seg.text.strip , } print f"language={info.language} segments={len lines }" The timestamps are the skeleton of the whole pipeline. Every downstream step preserves start / end per segment, because that's where the translated speech has to fit back. Per-segment MT gives you sentences that are individually fine and collectively wrong inconsistent terminology, drifting register . Feed the whole transcript to your translation step with context, and, crucially, give it a length constraint per segment. This is the single biggest lever against sync drift: dub/translate.py engine-agnostic sketch PROMPT = """Translate this video narration from English to Spanish. Rules: - Keep terminology consistent glossary: {glossary} - Each numbered line must be speakable within its duration. Line 3 has 2.8s. Line 7 has 4.1s. Prefer shorter phrasings. - Return the same numbered lines, translated.""" Whether the engine is an LLM, a local NLLB/M2M model, or a cloud MT API matters less than the contract: same segments in, same segments out, lengths respected. Have a native speaker skim the output. One reviewer-hour here prevents most of the embarrassment this pipeline can produce. XTTS-v2 supports 17 languages and clones a voice from a few seconds of clean reference audio. Cut a reference clip of the original narrator no music, no crosstalk : bash $ ffmpeg -i source.wav -ss 00:00:12 -t 8 -y reference.wav python dub/synthesize.py from TTS.api import TTS tts = TTS "tts models/multilingual/multi-dataset/xtts v2" for i, seg in enumerate translated segments : tts.tts to file text=seg "text es" , speaker wav="reference.wav", language="es", file path=f"segments/{i:04d}.wav", First run downloads the weights; after that it's local. GPU strongly recommended; CPU works for short content if you're patient. Spanish runs longer than English as a rule. Some synthesized segments will overflow their slots, and naive concatenation drifts out of sync within minutes. Measure first: python dub/align.py import soundfile as sf report = for i, seg in enumerate translated segments : audio, sr = sf.read f"segments/{i:04d}.wav" actual = len audio / sr slot = seg "end" - seg "start" report.append i, slot, actual, actual / slot for i, slot, actual, ratio in report: flag = "⚠️ OVERFLOW" if ratio 1.1 else "ok" print f"seg {i:04d} slot={slot:.2f}s synth={actual:.2f}s ratio={ratio:.2f} {flag}" seg 0007 slot=4.10s synth=5.23s ratio=1.28 ⚠️ OVERFLOW seg 0012 slot=2.80s synth=2.91s ratio=1.04 ok Then apply fixes in escalating order: atempo up to ~1.1 is usually imperceptible on speech; beyond that it sounds rushed: bash $ ffmpeg -i segments/0007.wav -filter:a "atempo=1.12" -y segments/0007 fit.wav Build the final track by placing each segment at its original start on a silent canvas, then remux against the untouched video stream: bash assemble placed segments into one track adelay per segment, amix , then: $ ffmpeg -i input.mp4 -i dubbed es.wav \ -map 0:v -map 1:a -c:v copy -shortest -y output es.mp4 -c:v copy matters: the video stream is never re-encoded, so the dub costs nothing in visual quality. Don't create tutorial es final v2.mp4 files. Mux the dub as an additional audio track and let the player expose a language menu: bash $ ffmpeg -i input.mp4 -i dubbed es.wav \ -map 0 -map 1:a -c copy \ -metadata:s:a:0 language=eng -metadata:s:a:1 language=spa \ -y output multilang.mp4 For HLS delivery, each language becomes an audio rendition in the master playlist; one video ladder, N audio tracks, and the player switches without a second stream. A short list from the failure modes this kind of pipeline reliably produces: av1 vulkan " in every language. Pre-process the script: expand numbers to words in the target language, and decide whether code identifiers stay English they should .