{"slug": "build-an-ai-dubbing-pipeline-faster-whisper-xtts-v2-ffmpeg", "title": "Build an AI dubbing pipeline: faster-whisper + XTTS-v2 + FFmpeg", "summary": "A developer built a script that takes an English video and produces a Spanish-narrated version with a cloned voice of the original speaker, using faster-whisper for transcription, an LLM for translation, XTTS-v2 for voice synthesis, and FFmpeg for audio replacement. The pipeline addresses the common problem of translated audio not fitting its time slot by enforcing length constraints per segment during translation.", "body_md": "## TL;DR\n\nWe're building a script that takes a video in English and produces the same video narrated in Spanish, in a cloned version of the original speaker's voice. Stack: faster-whisper for timestamped transcription, an LLM (or any MT engine) for translation, XTTS-v2 for voice-cloned synthesis, FFmpeg for surgery. We'll also handle the problem every demo skips: translated audio that doesn't fit its time slot.\n\n📦 Code: github.com/USER/repo (replace before publishing)\n\nIf you'd rather start from a finished system, Softcatala's [open-dubbing](https://github.com/softcatala/open-dubbing) and [KrillinAI](https://github.com/krillinai/KrillinAI) are full pipelines behind one CLI. This post builds the minimal version by hand so you understand what those tools are doing, and where they break.\n\nPython 3.10–3.12. The original Coqui company shut down in early 2024; the maintained fork of their TTS library is published by Idiap as `coqui-tts`\n\n:\n\n``` bash\n$ python -m venv dub && source dub/bin/activate\n$ pip install faster-whisper coqui-tts\n$ ffmpeg -version | head -1   # 6.0+ is fine, 8.x current\n```\n\n⚠️ Note: the XTTS-v2\n\nmodel weightsship under the Coqui Public Model License, which restricts commercial use. Prototype freely, but before dubbed videos ship to paying customers, someone must read that license and possibly swap the synthesis step for a commercially licensed model or paid API. Voice cloning also requires the speaker's consent. Get it in writing.\n\n``` bash\n# pull mono 16k audio for the ASR step\n$ ffmpeg -i input.mp4 -vn -ac 1 -ar 16000 -y source.wav\npython\n# dub/transcribe.py\nfrom faster_whisper import WhisperModel\n\nmodel = WhisperModel(\"large-v3-turbo\", compute_type=\"int8\")\nsegments, info = model.transcribe(\"source.wav\", word_timestamps=True)\n\nlines = []\nfor seg in segments:\n    lines.append({\n        \"start\": seg.start,\n        \"end\": seg.end,\n        \"text\": seg.text.strip(),\n    })\nprint(f\"language={info.language} segments={len(lines)}\")\n```\n\nThe timestamps are the skeleton of the whole pipeline. Every downstream step preserves `start`\n\n/`end`\n\nper segment, because that's where the translated speech has to fit back.\n\nPer-segment MT gives you sentences that are individually fine and collectively wrong (inconsistent terminology, drifting register). Feed the whole transcript to your translation step with context, and, crucially, give it a length constraint per segment. This is the single biggest lever against sync drift:\n\n```\n# dub/translate.py (engine-agnostic sketch)\nPROMPT = \"\"\"Translate this video narration from English to Spanish.\nRules:\n- Keep terminology consistent (glossary: {glossary})\n- Each numbered line must be speakable within its duration.\n  Line 3 has 2.8s. Line 7 has 4.1s. Prefer shorter phrasings.\n- Return the same numbered lines, translated.\"\"\"\n```\n\nWhether the engine is an LLM, a local NLLB/M2M model, or a cloud MT API matters less than the contract: same segments in, same segments out, lengths respected. Have a native speaker skim the output. One reviewer-hour here prevents most of the embarrassment this pipeline can produce.\n\nXTTS-v2 supports 17 languages and clones a voice from a few seconds of clean reference audio. Cut a reference clip of the original narrator (no music, no crosstalk):\n\n``` bash\n$ ffmpeg -i source.wav -ss 00:00:12 -t 8 -y reference.wav\npython\n# dub/synthesize.py\nfrom TTS.api import TTS\n\ntts = TTS(\"tts_models/multilingual/multi-dataset/xtts_v2\")\n\nfor i, seg in enumerate(translated_segments):\n    tts.tts_to_file(\n        text=seg[\"text_es\"],\n        speaker_wav=\"reference.wav\",\n        language=\"es\",\n        file_path=f\"segments/{i:04d}.wav\",\n    )\n```\n\nFirst run downloads the weights; after that it's local. GPU strongly recommended; CPU works for short content if you're patient.\n\nSpanish runs longer than English as a rule. Some synthesized segments will overflow their slots, and naive concatenation drifts out of sync within minutes. Measure first:\n\n``` python\n# dub/align.py\nimport soundfile as sf\n\nreport = []\nfor i, seg in enumerate(translated_segments):\n    audio, sr = sf.read(f\"segments/{i:04d}.wav\")\n    actual = len(audio) / sr\n    slot = seg[\"end\"] - seg[\"start\"]\n    report.append((i, slot, actual, actual / slot))\n\nfor i, slot, actual, ratio in report:\n    flag = \"⚠️ OVERFLOW\" if ratio > 1.1 else \"ok\"\n    print(f\"seg {i:04d}  slot={slot:.2f}s  synth={actual:.2f}s  ratio={ratio:.2f}  {flag}\")\nseg 0007  slot=4.10s  synth=5.23s  ratio=1.28  ⚠️ OVERFLOW\nseg 0012  slot=2.80s  synth=2.91s  ratio=1.04  ok\n```\n\nThen apply fixes in escalating order:\n\n`atempo`\n\nup to ~1.1 is usually imperceptible on speech; beyond that it sounds rushed:\n\n``` bash\n$ ffmpeg -i segments/0007.wav -filter:a \"atempo=1.12\" -y segments/0007_fit.wav\n```\n\nBuild the final track by placing each segment at its original `start`\n\non a silent canvas, then remux against the untouched video stream:\n\n``` bash\n# assemble placed segments into one track (adelay per segment, amix), then:\n$ ffmpeg -i input.mp4 -i dubbed_es.wav \\\n    -map 0:v -map 1:a -c:v copy -shortest -y output_es.mp4\n```\n\n`-c:v copy`\n\nmatters: the video stream is never re-encoded, so the dub costs nothing in visual quality.\n\nDon't create `tutorial_es_final_v2.mp4`\n\nfiles. Mux the dub as an additional audio track and let the player expose a language menu:\n\n``` bash\n$ ffmpeg -i input.mp4 -i dubbed_es.wav \\\n    -map 0 -map 1:a -c copy \\\n    -metadata:s:a:0 language=eng -metadata:s:a:1 language=spa \\\n    -y output_multilang.mp4\n```\n\nFor HLS delivery, each language becomes an audio rendition in the master playlist; one video ladder, N audio tracks, and the player switches without a second stream.\n\nA short list from the failure modes this kind of pipeline reliably produces:\n\n`av1_vulkan`\n\n\" in every language. Pre-process the script: expand numbers to words in the target language, and decide whether code identifiers stay English (they should).", "url": "https://wpnews.pro/news/build-an-ai-dubbing-pipeline-faster-whisper-xtts-v2-ffmpeg", "canonical_source": "https://dev.to/masonwritescode/build-an-ai-dubbing-pipeline-faster-whisper-xtts-v2-ffmpeg-114h", "published_at": "2026-07-08 06:29:58+00:00", "updated_at": "2026-07-08 06:58:40.351000+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "developer-tools"], "entities": ["faster-whisper", "XTTS-v2", "FFmpeg", "Coqui", "Idiap", "Softcatala", "KrillinAI", "NLLB"], "alternates": {"html": "https://wpnews.pro/news/build-an-ai-dubbing-pipeline-faster-whisper-xtts-v2-ffmpeg", "markdown": "https://wpnews.pro/news/build-an-ai-dubbing-pipeline-faster-whisper-xtts-v2-ffmpeg.md", "text": "https://wpnews.pro/news/build-an-ai-dubbing-pipeline-faster-whisper-xtts-v2-ffmpeg.txt", "jsonld": "https://wpnews.pro/news/build-an-ai-dubbing-pipeline-faster-whisper-xtts-v2-ffmpeg.jsonld"}}