# I Built a Hybrid AI / Spatial Video Upscaler in the Browser

> Source: <https://dev.to/hlinhbuilds/i-built-a-hybrid-ai-spatial-video-upscaler-in-the-browser-4o0>
> Published: 2026-09-02 13:01:04+00:00

Running [video upscaling](https://tapirconvert.com/video-upscaler) completely client-side in the browser sounds like a fun weekend project until you hit the reality of VRAM limits and Out-Of-Memory (OOM) tab crashes.

While building the video upscaler for my project tapirconvert, I realized that running a Heavy Neural Network (like Real-ESRGAN or ESPCN) to upscale a 1080p video to 4K natively in the browser is practically impossible for the average user's hardware.

So, I built a Hybrid Upscaling Pipeline that dynamically switches between ONNX AI Inference and Pure WebGPU Spatial Upscaling based on the input resolution. Here is how it works under the hood.

**🧠 The Architecture: The Resolution Router**

The core logic lies in the VideoUpscaler.svelte UI component. Before processing begins, we probe the video metadata. The decision matrix is simple but effective:

`// If the video is 720p or larger, AI is too expensive. Use FSR (Spatial).`

// If it's SD, use IMDN (AI).

const isHD = Math.max(w, h) >= 1280 && Math.min(w, h) >= 720;

worker = isHD ? new UpscaleWorker() : new ImdnWorker();

By decoupling the workers, we guarantee that we apply the right tool for the job without melting the user's GPU.

**🧪 Pipeline 1: The SD Path (AI via IMDN / ONNX Runtime)**

For low-resolution videos (e.g., 360p or 480p), traditional upscaling algorithms just make the blur bigger. We need the AI to "hallucinate" and reconstruct lost details.

The Model: I use an IMDN (Information Multi-distillation Network) architecture via onnxruntime-web.

The Execution: The model runs inside a Web Worker. Because the input resolution is small, the tensor allocations (even when batched or padded) easily fit within standard WebGPU limits (usually 128MB max buffer size). The AI does a fantastic job of reconstructing sharp edges, hair, and textures.

**⚡ Pipeline 2: The HD Path (AMD FSR 1.0 via WGSL Shaders)**

When a user uploads a 1080p video and wants 4K, running that through an ONNX model in a browser tab takes seconds per frame and usually crashes.

Instead of AI, I implemented AMD's FSR 1.0 (FidelityFX Super Resolution) entirely in custom WGSL Compute Shaders, completely bypassing ONNX.

Since this is pure GPU math running natively via WebGPU, it processes 1080p -> 4K frames in milliseconds. It’s not AI, but at HD resolutions, the pixel density is already high enough that a high-quality spatial upscaler looks nearly identical to AI, but at 1/100th the compute cost.

**🛠️ The Data Flow (Avoiding CPU/GPU ping-pong)**

In both pipelines, the biggest bottleneck isn't the math; it's moving data between the CPU (JavaScript) and the GPU. I use mediabunny (a wrapper over WebCodecs/FFmpeg) to extract VideoFrame objects. These are passed directly to WebGPU using copyExternalImageToTexture. The processing happens entirely in VRAM, and we only map the final output buffer back to the CPU to paint it onto an OffscreenCanvas before re-encoding it into an MP4 container.

**✅ The Takeaway**

If you are building client-side AI tools, don't blindly throw Neural Networks at every problem. Fallback mechanisms are your best friend. Using AI where it matters (low-res) and blazing-fast GPU shaders where it doesn't (high-res) is the only way to build a production-ready browser upscaler today.

Has anyone else played around with porting FSR or DLSS-like shaders to WebGPU? Let’s chat in the comments!