# I built a local-first AI video dubbing pipeline with WhisperX + VoxCPM2

> Source: <https://dev.to/kadirb4rut/i-built-a-local-first-ai-video-dubbing-pipeline-with-whisperx-voxcpm2-1bek>
> Published: 2026-08-19 10:28:11+00:00

I wanted to dub short videos without sending the source media to a hosted dubbing platform. The hard part is not producing one synthetic sentence. It is keeping the original speech timing, preserving the background, translating the transcript, cloning a voice, and rebuilding a video that still feels like the same video.

That led me to build [Video Dubbing Translator](https://github.com/kadirb4rut/video-dubbing-translator), a local-first, browser-based pipeline for short-form video dubbing.

The current flow is:

**video → vocal/background separation → Whisper/WhisperX → translation → VoxCPM2 voice cloning → duration fitting → reconstruction → optional LatentSync**

The interface is designed around inspectable stages rather than one opaque “dub this” button. I can keep the extracted vocal and background tracks separate, see the transcript and translation, synthesize each translated segment, fit its duration, and then rebuild the final video.

Translation currently uses Google Translate, so “local-first” does not mean “fully offline.” The media processing and speech synthesis run locally, while translation is the deliberate network-dependent step.

The project previously used XTTS-v2. I migrated the active voice-cloning path completely to VoxCPM2 because it is the direction I want this project to grow around: reference-voice cloning, native 48 kHz speech output, and an official model/source with an Apache-2.0 license.

This is not a claim that one model wins every voice or language test. It is a practical integration decision. VoxCPM2 gives the pipeline a clear voice-cloning backend, and the 48 kHz output fits naturally into the reconstruction stage.

I ran an end-to-end test on a 15-second English clip:

The rendered output is about 14.88 seconds long. Running Turkish ASR over the final audio recovered all four intended Turkish sentences. That is useful evidence that the pipeline works end to end, but it is one short integration test—not a scientific benchmark.

Here are the artifacts:

A translated sentence rarely has the same duration as the source sentence. I therefore treat duration fitting as its own stage instead of hoping the synthesizer happens to match. The target duration comes from the timed transcript, and the generated segment is adjusted before reconstruction.

For a short clip, the background is part of the identity of the scene. Separating it before synthesis means the new speech can replace the original vocal track without throwing away the surrounding sound.

The current demo is rebuilt with 48 kHz stereo audio and H.264/AAC video. Making the output contract explicit catches format problems early and makes the resulting file easier to inspect or share.

LatentSync is an optional stage, not a hidden requirement of the base pipeline. The core demo is speech dubbing and reconstruction; lip-sync can be added when the project and hardware are ready for that extra pass.

There are several important boundaries:

Those limits are part of why I am publishing the implementation and the artifacts instead of presenting the demo as a finished dubbing product.

The repository is MIT-licensed and includes the browser UI, processing stages, setup notes, and the demo links:

[github.com/kadirb4rut/video-dubbing-translator](https://github.com/kadirb4rut/video-dubbing-translator)

I would especially like feedback on voice similarity, timing behavior, installation on different GPUs, multilingual edge cases, and how the pipeline behaves with more than one speaker. The goal is a reproducible local-first workflow that people can inspect and improve—not a black-box hosted service.

If you are building with WhisperX, VoxCPM2, or local video tooling, I would be interested in what you would change first.