Create an AI Dubbing Tool for Video Content A developer published a step-by-step guide for building a lightweight AI dubbing pipeline in Python that extracts audio from video with ffmpeg, optionally transcribes it with Whisper, synthesizes a new voice track through the ElevenLabs text-to-speech REST API, and time-stretches or pads the result with pydub before muxing it back onto the footage. The walkthrough includes working code for each stage and notes that cloned voice IDs can be swapped in for stock voices. If you’ve ever tried to localize a video, you know the pain: hiring voice actors, coordinating recording sessions, and then painstakingly syncing the new audio back to the footage. With modern text‑to‑speech TTS and voice‑cloning services, you can automate most of that workflow. In this article we’ll build a lightweight AI dubbing pipeline that takes a source video, generates a new voice track in any language you like, and stitches it back together—all with a few lines of Python and a call to ElevenLabs https://try.elevenlabs.io/kr07zfuqn1bp . TL;DR – Extract audio → transcribe optional → generate new speech → align & mux → 🎉 | Item | Reason | |---|---| | Python 3.9+ | Main scripting language | | ffmpeg | Extract and re‑encode audio/video | | pydub | Simple audio manipulation | | requests | Call ElevenLabs REST API | | Optional whisper or any speech‑to‑text model | If you want to translate the script before dubbing | | An ElevenLabs API key | The magic behind high‑quality voice synthesis | Install the Python dependencies: pip install pydub requests tqdm optional for transcription pip install openai-whisper Make sure ffmpeg is in your PATH – most OS package managers have it brew install ffmpeg , apt-get install ffmpeg , etc. . First we need the raw audio track from the video. ffmpeg can do this in a single command: ffmpeg -i input video.mp4 -vn -acodec pcm s16le -ar 44100 -ac 2 original.wav -vn disables video output. pcm s16le gives us a lossless WAV file that works well with most TTS services. If you prefer to keep everything inside Python, subprocess works just as well: python import subprocess def extract audio video path, wav path : cmd = "ffmpeg", "-i", video path, "-vn", "-acodec", "pcm s16le", "-ar", "44100", "-ac", "2", wav path, "-y" overwrite if exists subprocess.run cmd, check=True extract audio "input video.mp4", "original.wav" If you already have a script, skip this step. Otherwise, you can run an automatic transcription and then translate it with any LLM or translation API. Here’s a quick Whisper example: python import whisper model = whisper.load model "base" result = model.transcribe "original.wav" script = result "text" print "Transcribed script:", script You could pipe script into Google Translate, DeepL, or even an OpenAI LLM to get the target language text. ElevenLabs offers a REST endpoint that can produce lifelike speech in a matter of seconds. The following function sends the text and streams back a WAV file. python import requests from pathlib import Path ELEVEN API KEY = "YOUR ELEVENLABS API KEY" VOICE ID = "EXAVITQu4vr4xnSDxMaL" default “Rachel” voice – replace with your cloned voice ID def synthesize text: str, out path: Path : url = f"https://api.elevenlabs.io/v1/text-to-speech/{VOICE ID}" headers = { "xi-api-key": ELEVEN API KEY, "Content-Type": "application/json", "Accept": "audio/wav" } payload = { "text": text, "model id": "eleven monolingual v1", "voice settings": { "stability": 0.75, "similarity boost": 0.85 } } response = requests.post url, json=payload, headers=headers, stream=True response.raise for status with open out path, "wb" as f: for chunk in response.iter content chunk size=8192 : f.write chunk synthesize script, Path "dub.wav" Pro tip: If you’ve trained a custom voice on ElevenLabs they call it voice cloning , just replace VOICE ID with the ID you received after the cloning process. The same API call works for both stock and cloned voices. The naive approach is to assume the new script has the same duration as the original. In practice, you’ll need to time‑stretch or pad the audio to match the video’s timeline. pydub makes this straightforward: python from pydub import AudioSegment from pydub.utils import mediainfo def match duration source wav, target wav : src = AudioSegment.from wav source wav tgt = AudioSegment.from wav target wav src len = len src milliseconds tgt len = len tgt if tgt len src len: Trim excess tgt = tgt :src len else: Pad with silence at the end silence = AudioSegment.silent duration=src len - tgt len tgt = tgt + silence tgt.export "dub aligned.wav", format="wav" return "dub aligned.wav" aligned path = match duration "original.wav", "dub.wav" If you need finer control e.g., per‑sentence alignment , you can use forced‑alignment tools like Aeneas or Montreal Forced Aligner , but that’s beyond the scope of this quick demo. Now that the audio length matches the video, we simply replace the original track: ffmpeg -i input video.mp4 -i dub aligned.wav -c:v copy -map 0:v:0 -map 1:a:0 -shortest output dubbed.mp4 -c:v copy tells ffmpeg to keep the video stream unchanged. -shortest ensures the output stops when the shortest stream ends useful if you have tiny mismatches . You can wrap this in Python as well: python def mux video path, audio path, out path : cmd = "ffmpeg", "-i", video path, "-i", audio path, "-c:v", "copy", "-map", "0:v:0", "-map", "1:a:0", "-shortest", out path, "-y" subprocess.run cmd, check=True mux "input video.mp4", aligned path, "output dubbed.mp4" That’s it—your video now speaks in a brand‑new voice Putting everything together in a single script makes it reusable: python def dub video video file: str, target text: str, output file: str, voice id: str = VOICE ID, : extract audio video file, "original.wav" synthesize target text, Path "dub.wav" aligned = match duration "original.wav", "dub.wav" mux video file, aligned, output file Example usage dub video video file="input video.mp4", target text=script, could be the original script or a translated version output file="final dub.mp4" Feel free to expand it with command‑line arguments, batch processing, or a tiny Flask API if you want a web‑based dubbing service.