You can explore the source code and experiment with the pipeline here:
GitHub — Sumit884-byte/dub_movie
What if you could take a Japanese movie or anime, translate its dialogue into Hindi, generate new speech automatically, preserve the background audio, and produce a finished dubbed video—all from a Python pipeline?
That is the idea behind dub_movie, an automated movie-dubbing project built around Python, Whisper, Edge TTS, translation backends, audio analysis, and FFmpeg.
The project is particularly designed for anime-style content with named characters and subtitle files, while also providing tools for processing longer videos in resumable stages.
dub_movie takes a source video and subtitles and turns them into a dubbed version in another language.
The basic workflow looks like this:
Japanese Movie / Anime
│
▼
Subtitle / Audio
│
▼
Speech Detection
(Whisper)
│
▼
Speaker Identification
│
▼
Voice / Pitch Analysis
│
▼
Translation
│
▼
Edge TTS
│
▼
Background Audio
+ Dubbed Speech
│
▼
Final MP4 Video
The repository describes its primary use case as Japanese source video plus SRT subtitles, producing translated speech with Edge TTS while matching character pitch and mixing the generated dialogue over a vocal-reduced background.
At first glance, dubbing sounds simple:
Speech → Translation → Text-to-Speech
But real video dubbing has several additional problems.
The translated sentence needs to appear at the correct timestamp. The generated speech needs to fit inside the original dialogue duration. Different characters should sound different. Background music and sound effects should remain audible.
This means a useful dubbing system needs to solve several problems simultaneously:
The dub_movie project approaches these as separate pipeline stages instead of treating dubbing as one giant operation.
The project is primarily Python-based and requires Python 3.10+ and FFmpeg. GPU acceleration is optional; Whisper can run on CPU, while local NLLB translation can benefit from CUDA.
| Technology | Role |
|---|---|
| Python | Main application and orchestration |
| Faster-Whisper | Speech recognition and timing |
| Edge TTS | Text-to-speech generation |
| MoviePy | Video/audio processing |
| Librosa | Pitch/F0 analysis |
| NumPy | Numerical audio processing |
| FFmpeg | Media extraction and rendering |
| Deep Translator | Google translation backend |
| Transformers + NLLB | Optional local translation |
| Demucs | Optional higher-quality vocal separation |
| vLLM | Optional speaker identification |
The core installation includes edge-tts, moviepy, numpy, librosa, httpx, deep-translator, and faster-whisper.
One of the important components is Faster-Whisper.
Instead of simply translating the subtitles, the project can analyze the actual speech in the video and use Whisper to determine when dialogue occurs.
The recommended dub_movie_whisper.py pipeline combines Whisper speech detection with SRT alignment, making it possible to adjust subtitle timing around the actual spoken dialogue.
For example:
python dub_movie_whisper.py \
--input movie.mp4 \
--srt movie.srt \
--output movie_dubbed_1min.mp4 \
--duration 60 \
--language hi
Here, hi represents Hindi.
The system also supports different Whisper model sizes such as:
tiny
base
small
medium
This creates a practical trade-off between processing speed and recognition quality.
A normal translation system might translate every sentence independently.
That can be a problem in anime or movies.
Characters have different personalities, relationships, speech patterns, and tones. A sentence that sounds natural for one character might sound completely wrong for another.
The repository therefore includes character_context.py, which is responsible for cast profiles, tone detection, contextual translation, and TTS voice mapping.
This is an important idea:
Movie dubbing is not just language translation. It is contextual translation.
For example, a character profile can help distinguish between:
"Come here."
spoken by:
The literal translation may be similar, but the delivery should not be.
The project can optionally use an OpenAI-compatible vLLM endpoint to identify which character is speaking each subtitle line.
The relevant configuration includes:
VLLM_BASE_URL
VLLM_MODEL
VLLM_API_KEY
VLLM_SPEAKER_BATCH
VLLM_TIMEOUT
The speaker-identification module is implemented in speaker_id_vllm.py.
This becomes especially useful when a subtitle file contains dialogue from multiple characters but doesn't reliably identify the speaker.
One of the more interesting parts of the project is its voice analysis stage.
The system uses Librosa to estimate the fundamental frequency, or F0, of speech segments.
That information can then influence the Edge TTS rate and pitch settings.
The goal isn't necessarily to clone someone's exact voice. Instead, the system tries to make generated speech better match the characteristics of the original speaker.
The pipeline describes this as:
Original dialogue
↓
F0 analysis
↓
Character voice profile
↓
Edge TTS rate/pitch
↓
Generated dialogue
This is a clever middle ground between basic text-to-speech and full voice cloning.
Simply removing the original audio and replacing it with generated speech would destroy the movie's atmosphere.
Background music, ambience, and sound effects are a huge part of the viewing experience.
The project therefore creates a vocal-reduced background bed.
Its pipeline can use channel routing and optionally Demucs for higher-quality vocal separation.
The result is approximately:
Original Video
│
├── Dialogue ──X
│
└── Background ─────┐
│
Generated Hindi Speech ──┤
▼
Final Audio
The final audio is then combined with the original video.
Another useful feature is that translation isn't hardcoded to a single provider.
The project supports several translation modes, including:
auto
parallel
minimax
cloud
google
The auto mode can use multiple backends in parallel, while Google translation is available through deep-translator.
There is also an offline NLLB option using Transformers:
facebook/nllb-200-distilled-600M
The device can be configured for CPU or CUDA.
That makes the architecture more flexible:
Translation
│
┌──────────┼──────────┐
▼ ▼ ▼
Google Cloud/LLM NLLB
Developers can choose between convenience, speed, cost, and local processing.
Movie dubbing is computationally expensive.
If you're processing a two-hour movie and the process crashes near the end, restarting everything from zero would be painful.
dub_movie addresses this with state files and intermediate caches.
The pipeline records stages such as:
start
↓
whisper_audio_ready
↓
whisper_done
↓
aligned
↓
speakers_vllm
↓
voice_analyzed
↓
translating
↓
translated
↓
background_ready
↓
synthesizing
↓
synthesized
↓
rendering
↓
done
The project stores files such as:
*.state.json
translated_segments_*.json
*.voice.json
*.speakers.json
temp_chunks_*/
*.background.wav
These artifacts allow completed work to be reused during subsequent runs.
For a long movie, this is not a cosmetic feature. It's the difference between a usable pipeline and a frustrating one.
The repository provides several entry points depending on the task.
For a one-minute test:
python dub_movie_whisper.py \
--input movie.mp4 \
--srt movie.srt \
--output movie_dubbed_1min.mp4 \
--duration 60 \
--language hi
If the SRT timestamps are already correct:
python dub_movie_whisper.py \
--srt-timing-only \
--duration 60 \
--language hi
python dub_movie.py \
--input movie.mp4 \
--srt movie.srt \
--output movie_dubbed.mp4 \
--language hi \
--duration 300
The repository also includes preset scripts for 1-minute, 5-minute, 10-minute, clip, and full-movie processing.
This type of technology has applications beyond simply watching anime in another language.
Japanese anime can be converted into languages such as Hindi, English, Spanish, and others.
Creators can produce versions of videos for different language audiences without recording every language manually.
Courses and tutorials can potentially be converted into regional languages.
Independent filmmakers can experiment with alternate-language versions of their work.
Automatically generated speech can help make video content accessible to audiences who don't understand the original language.
The project is also a useful playground for experimenting with:
The biggest takeaway from dub_movie isn't any individual AI model.
It's the pipeline architecture.
Instead of asking one model to "dub this movie," the project breaks the problem into specialized stages:
VIDEO
│
▼
Audio Prep
│
┌─────────┴─────────┐
▼ ▼
Whisper SRT
│ │
└─────────┬─────────┘
▼
Alignment
│
▼
Speaker Mapping
│
▼
Voice Analysis
│
▼
Translation
│
▼
TTS
│
▼
Audio Composition
│
▼
FFmpeg
│
▼
FINAL VIDEO
Each stage can be improved independently.
Want better transcription? Change the ASR model.
Want better translation? Change the translation backend.
Want better voice quality? Replace the TTS engine.
Want better vocal separation? Use Demucs.
That modularity is what makes this kind of project interesting from an engineering perspective.
The current architecture also reveals several areas for future development.
The project currently focuses on pitch/rate matching with Edge TTS rather than full speaker voice cloning.
A future version could integrate modern voice-cloning models to produce more character-specific voices.
The current pipeline focuses primarily on audio timing. A more advanced system could modify facial animation or lip movements to match translated speech.
Context-aware translation is already part of the architecture, but dialogue-heavy content could benefit from larger contextual windows covering entire scenes.
Speaker identification could potentially combine audio diarization with subtitle context and LLM reasoning.
Long movies involve thousands of speech segments. More aggressive batching, GPU inference, and caching could significantly reduce processing time.
dub_movie is particularly interesting for developers who want to learn how several AI technologies can be connected into a single real-world media pipeline.
It isn't just a chatbot project.
It combines:
Speech Recognition + Translation + LLMs + Text-to-Speech + Signal Processing + Audio Separation + Video Processing.
That's what makes it a useful engineering project.
AI dubbing is moving from a theoretical idea toward an increasingly practical media-processing workflow.
The dub_movie project demonstrates one way to build that workflow using open development tools and interchangeable components.
Its most interesting feature isn't simply generating Hindi speech. The real engineering challenge is coordinating timing, translation, characters, pitch, background audio, caching, and final video rendering into one repeatable pipeline.
For developers interested in AI-powered media applications, this project provides a practical example of how multiple specialized technologies can be assembled into a complete application.
Note: The repository is intended for processing video that you have the rights to use. Copyright and voice/likeness rights can apply to both source movies and generated dubbing.