Watermark and unwanted-object removal is one of those features that looks simple in a model demo and becomes much more complicated when you put it inside a real video editor.
The obvious implementation is to upload the media to a GPU server, run an inpainting model, and return the result. That works, but it introduces upload latency, infrastructure cost, and privacy concerns—especially for large or unpublished videos.
For our browser video editor, we took a different approach:
Run MI-GAN locally with ONNX Runtime Web and WebGPU, so images and videos never need to leave the user's device.
This article covers more than model inference. It explains the product-level work required for multiple repair regions, time ranges, moving watermarks, progressive frame previews, before/after comparison, cancellation, and final video encoding.
Use object-removal tools only on media you own or are authorized to edit. Removing ownership or attribution marks from third-party content may violate licenses or platform rules.
A server-side pipeline usually requires:
Browser-local inference moves the initial cost to down the model. Once the runtime and model artifacts are cached, repeated edits are much faster to start, and the original media remains local.
Our implementation uses:
Image repair is a single inference task. Video repair has a time axis, so the pipeline is substantially different:
Source media
-> user defines repair regions and time ranges
-> resolve regions active at the current timestamp
-> decode a video frame
-> build a mask
-> run MI-GAN inference
-> composite the repaired pixels over the source frame
-> commit the progressive preview
-> encode the processed frame
-> create a new media asset
The model is only one stage. A production-quality experience also requires the canvas, playhead, processed-frame count, and progress indicator to share the same clock.
In image mode, the user draws one or more rectangles over the canvas. We store each rectangle in normalized coordinates so the selection remains correct across different preview sizes.
function createMask(width, height, regions) {
const canvas = new OffscreenCanvas(width, height);
const context = canvas.getContext("2d");
context.fillStyle = "#000";
context.fillRect(0, 0, width, height);
context.fillStyle = "#fff";
for (const region of regions) {
context.fillRect(
region.x * width,
region.y * height,
region.width * width,
region.height * height
);
}
return context.getImageData(0, 0, width, height);
}
Each region uses values between 0
and 1
. The same rectangle can therefore be mapped back to the full-resolution source instead of the CSS-sized preview.
After inference, we do not immediately commit the result. The editor displays a draggable vertical comparison line:
<ComparisonView
before={sourceUrl}
after={repairedUrl}
position={comparePosition}
onPositionChange={setComparePosition}
/>
That distinction matters because inpainting does not recover a hidden ground truth. It synthesizes plausible pixels from the surrounding context, so users need a clear way to inspect the result before applying it.
Our first video prototype repaired only the frame currently visible in the editor. It looked correct during a single-frame test but was obviously wrong for a video.
A watermark exists for a time range, not for one frame. It may also move: for example, it can appear in the bottom-left corner for the first few seconds and later switch to the top-right.
We represent every repair region as an independent time-scoped object:
const repairRegion = {
id: "region-1",
startTime: 0,
endTime: 4,
keyframes: [
{ time: 0, x: 0.72, y: 0.82, width: 0.22, height: 0.10 },
{ time: 4, x: 0.08, y: 0.08, width: 0.22, height: 0.10 }
]
};
The rectangle between two recorded positions is interpolated:
function interpolateRegion(a, b, ratio) {
return {
x: a.x + (b.x - a.x) * ratio,
y: a.y + (b.y - a.y) * ratio,
width: a.width + (b.width - a.width) * ratio,
height: a.height + (b.height - a.height) * ratio
};
}
This supports fixed watermarks, watermarks that jump to another corner, and slowly moving overlays without creating a separate repair job for every frame.
Real media may contain a logo and a text overlay at the same time. Both image and video workflows therefore need multiple selections.
At each video timestamp, we resolve all active regions and combine them into one mask:
const activeRegions = regions
.filter((region) => {
return time >= region.startTime && time <= region.endTime;
})
.map((region) => resolveRegionAtTime(region, time));
This data model also gives us a practical editing interface:
Creating the model session or running inference on the main thread can freeze the entire editor. We keep a long-lived model session inside a Web Worker and transfer frame data to it.
const session = await ort.InferenceSession.create(modelUrl, {
executionProviders: ["webgpu", "wasm"]
});
WebGPU is preferred. If it is unavailable or initialization fails, we explicitly fall back to WASM and report the actual backend in the UI.
worker.postMessage(
{
type: "repair",
requestId,
imageBitmap,
regions
},
[imageBitmap]
);
The Worker and inference session remain alive for later requests. Model and runtime assets are stored in a versioned Cache Storage entry, so a second repair should never present itself as another model download.
Inpainting models commonly expect a fixed input size. Resizing and regenerating the entire video frame for a small watermark can unintentionally change faces, text, and fine texture outside the selected area.
Our processing rules are:
context.drawImage(originalFrame, 0, 0);
context.save();
context.globalCompositeOperation = "source-over";
context.drawImage(
repairedPatch,
patchX,
patchY,
patchWidth,
patchHeight
);
context.restore();
Keeping regions reasonably tight improves both performance and output stability.
An early version used three independent notions of time:
The result was confusing: the progress bar might show 40% while the preview was still near 25%.
The fix was to make the committed visible frame the single UI clock:
for (let frameIndex = 0; frameIndex < totalFrames; frameIndex += 1) {
const frame = await decodeFrame(frameIndex);
const repairedFrame = await repairFrameIfNeeded(frame, frameIndex);
await onFrameCommitted(repairedFrame, frameIndex);
reportProgress({
completedFrames: frameIndex + 1,
totalFrames
});
}
Frames outside every repair range must still call onFrameCommitted
. Otherwise, the preview and playhead appear frozen whenever inference is skipped.
We also decode the progressive preview image and give the browser one paint before advancing the completed-frame count:
await image.decode();
setPreviewFrame(image.src);
await new Promise(requestAnimationFrame);
setCompletedFrame(frameIndex + 1);
During processing, the left side now displays the source frame for the current timestamp while the right side displays the repaired frame for that same timestamp. The comparison line stays interactive throughout the process.
Reaching 100% repaired frames does not mean the output file is ready. We still need to initialize the encoder, encode the frames, and create the final media asset.
We expose these as distinct phases:
1. Preparing the model
2. Repairing frames
3. the video encoder
4. Encoding the repaired video
5. Creating the new asset
6. Complete
Trying to compress all of these into a single unexplained percentage makes the UI appear stuck at the end. A phase label tells the user what the browser is actually doing.
For long videos, cancellation is not optional. Changing the button text to "Cancelling..." is not enough.
A real cancel path must:
VideoFrame
and ImageBitmap
objects.
if (abortSignal.aborted) {
throw new DOMException("Repair cancelled", "AbortError");
}
We disable duplicate cancel requests immediately, but return to an editable state only after cleanup has completed. The original media remains unchanged until the user explicitly applies the new result.
Our first UI placed all controls in the editor's right-side property panel. That panel quickly became too small for region drawing, time ranges, keyframes, preview, comparison, and progress.
The final product keeps AI Repair as a lightweight capability entry point and opens the actual work in a larger modal:
The mobile layout uses the same mental model in a vertical workspace. Applying closes both the repair modal and the properties drawer; cancelling closes only the modal.
Browser AI performance varies widely with GPU, resolution, codec, and browser implementation. Rather than promising a universal frame rate, we focused on these principles:
The first model load is still a cost, but cached repeat operations become a much better fit for an interactive editor.
MI-GAN is lightweight and fast enough to be practical in a browser, but it is not perfect:
Frame-by-frame inpainting does not guarantee temporal consistency. Optical flow, region tracking, and temporally aware restoration models are natural next steps.
The biggest lesson from building this feature was:
Successful model inference is only the beginning. The real product is the editing flow that lets users inspect, trust, cancel, and apply the result.
Repairing one image in a demo and shipping multi-region, time-aware, progressively previewed video repair are completely different engineering problems.
This implementation runs locally in the browser and inserts the result as a new media asset, leaving the original recoverable.
If you are experimenting with WebGPU, ONNX Runtime Web, browser-side media processing, or video-editor UX, I hope these implementation details save you a few iterations.