# Build an AI dubbing pipeline: faster-whisper + XTTS-v2 + FFmpeg

> Source: <https://dev.to/masonwritescode/build-an-ai-dubbing-pipeline-faster-whisper-xtts-v2-ffmpeg-114h>
> Published: 2026-07-08 06:29:58+00:00

## TL;DR

We're building a script that takes a video in English and produces the same video narrated in Spanish, in a cloned version of the original speaker's voice. Stack: faster-whisper for timestamped transcription, an LLM (or any MT engine) for translation, XTTS-v2 for voice-cloned synthesis, FFmpeg for surgery. We'll also handle the problem every demo skips: translated audio that doesn't fit its time slot.

📦 Code: github.com/USER/repo (replace before publishing)

If you'd rather start from a finished system, Softcatala's [open-dubbing](https://github.com/softcatala/open-dubbing) and [KrillinAI](https://github.com/krillinai/KrillinAI) are full pipelines behind one CLI. This post builds the minimal version by hand so you understand what those tools are doing, and where they break.

Python 3.10–3.12. The original Coqui company shut down in early 2024; the maintained fork of their TTS library is published by Idiap as `coqui-tts`

:

``` bash
$ python -m venv dub && source dub/bin/activate
$ pip install faster-whisper coqui-tts
$ ffmpeg -version | head -1   # 6.0+ is fine, 8.x current
```

⚠️ Note: the XTTS-v2

model weightsship under the Coqui Public Model License, which restricts commercial use. Prototype freely, but before dubbed videos ship to paying customers, someone must read that license and possibly swap the synthesis step for a commercially licensed model or paid API. Voice cloning also requires the speaker's consent. Get it in writing.

``` bash
# pull mono 16k audio for the ASR step
$ ffmpeg -i input.mp4 -vn -ac 1 -ar 16000 -y source.wav
python
# dub/transcribe.py
from faster_whisper import WhisperModel

model = WhisperModel("large-v3-turbo", compute_type="int8")
segments, info = model.transcribe("source.wav", word_timestamps=True)

lines = []
for seg in segments:
    lines.append({
        "start": seg.start,
        "end": seg.end,
        "text": seg.text.strip(),
    })
print(f"language={info.language} segments={len(lines)}")
```

The timestamps are the skeleton of the whole pipeline. Every downstream step preserves `start`

/`end`

per segment, because that's where the translated speech has to fit back.

Per-segment MT gives you sentences that are individually fine and collectively wrong (inconsistent terminology, drifting register). Feed the whole transcript to your translation step with context, and, crucially, give it a length constraint per segment. This is the single biggest lever against sync drift:

```
# dub/translate.py (engine-agnostic sketch)
PROMPT = """Translate this video narration from English to Spanish.
Rules:
- Keep terminology consistent (glossary: {glossary})
- Each numbered line must be speakable within its duration.
  Line 3 has 2.8s. Line 7 has 4.1s. Prefer shorter phrasings.
- Return the same numbered lines, translated."""
```

Whether the engine is an LLM, a local NLLB/M2M model, or a cloud MT API matters less than the contract: same segments in, same segments out, lengths respected. Have a native speaker skim the output. One reviewer-hour here prevents most of the embarrassment this pipeline can produce.

XTTS-v2 supports 17 languages and clones a voice from a few seconds of clean reference audio. Cut a reference clip of the original narrator (no music, no crosstalk):

``` bash
$ ffmpeg -i source.wav -ss 00:00:12 -t 8 -y reference.wav
python
# dub/synthesize.py
from TTS.api import TTS

tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2")

for i, seg in enumerate(translated_segments):
    tts.tts_to_file(
        text=seg["text_es"],
        speaker_wav="reference.wav",
        language="es",
        file_path=f"segments/{i:04d}.wav",
    )
```

First run downloads the weights; after that it's local. GPU strongly recommended; CPU works for short content if you're patient.

Spanish runs longer than English as a rule. Some synthesized segments will overflow their slots, and naive concatenation drifts out of sync within minutes. Measure first:

``` python
# dub/align.py
import soundfile as sf

report = []
for i, seg in enumerate(translated_segments):
    audio, sr = sf.read(f"segments/{i:04d}.wav")
    actual = len(audio) / sr
    slot = seg["end"] - seg["start"]
    report.append((i, slot, actual, actual / slot))

for i, slot, actual, ratio in report:
    flag = "⚠️ OVERFLOW" if ratio > 1.1 else "ok"
    print(f"seg {i:04d}  slot={slot:.2f}s  synth={actual:.2f}s  ratio={ratio:.2f}  {flag}")
seg 0007  slot=4.10s  synth=5.23s  ratio=1.28  ⚠️ OVERFLOW
seg 0012  slot=2.80s  synth=2.91s  ratio=1.04  ok
```

Then apply fixes in escalating order:

`atempo`

up to ~1.1 is usually imperceptible on speech; beyond that it sounds rushed:

``` bash
$ ffmpeg -i segments/0007.wav -filter:a "atempo=1.12" -y segments/0007_fit.wav
```

Build the final track by placing each segment at its original `start`

on a silent canvas, then remux against the untouched video stream:

``` bash
# assemble placed segments into one track (adelay per segment, amix), then:
$ ffmpeg -i input.mp4 -i dubbed_es.wav \
    -map 0:v -map 1:a -c:v copy -shortest -y output_es.mp4
```

`-c:v copy`

matters: the video stream is never re-encoded, so the dub costs nothing in visual quality.

Don't create `tutorial_es_final_v2.mp4`

files. Mux the dub as an additional audio track and let the player expose a language menu:

``` bash
$ ffmpeg -i input.mp4 -i dubbed_es.wav \
    -map 0 -map 1:a -c copy \
    -metadata:s:a:0 language=eng -metadata:s:a:1 language=spa \
    -y output_multilang.mp4
```

For HLS delivery, each language becomes an audio rendition in the master playlist; one video ladder, N audio tracks, and the player switches without a second stream.

A short list from the failure modes this kind of pipeline reliably produces:

`av1_vulkan`

" in every language. Pre-process the script: expand numbers to words in the target language, and decide whether code identifiers stay English (they should).