{"slug": "i-built-a-hybrid-ai-spatial-video-upscaler-in-the-browser", "title": "I Built a Hybrid AI / Spatial Video Upscaler in the Browser", "summary": "A developer built a hybrid video upscaler that runs entirely in the browser, dynamically switching between AI-based ONNX inference and WebGPU spatial upscaling based on input resolution. The tool, part of the tapirconvert project, uses IMDN neural networks for low-resolution videos and AMD's FSR 1.0 shaders for HD content to avoid VRAM crashes and maintain performance.", "body_md": "Running [video upscaling](https://tapirconvert.com/video-upscaler) completely client-side in the browser sounds like a fun weekend project until you hit the reality of VRAM limits and Out-Of-Memory (OOM) tab crashes.\n\nWhile building the video upscaler for my project tapirconvert, I realized that running a Heavy Neural Network (like Real-ESRGAN or ESPCN) to upscale a 1080p video to 4K natively in the browser is practically impossible for the average user's hardware.\n\nSo, I built a Hybrid Upscaling Pipeline that dynamically switches between ONNX AI Inference and Pure WebGPU Spatial Upscaling based on the input resolution. Here is how it works under the hood.\n\n**🧠 The Architecture: The Resolution Router**\n\nThe core logic lies in the VideoUpscaler.svelte UI component. Before processing begins, we probe the video metadata. The decision matrix is simple but effective:\n\n`// If the video is 720p or larger, AI is too expensive. Use FSR (Spatial).`\n\n// If it's SD, use IMDN (AI).\n\nconst isHD = Math.max(w, h) >= 1280 && Math.min(w, h) >= 720;\n\nworker = isHD ? new UpscaleWorker() : new ImdnWorker();\n\nBy decoupling the workers, we guarantee that we apply the right tool for the job without melting the user's GPU.\n\n**🧪 Pipeline 1: The SD Path (AI via IMDN / ONNX Runtime)**\n\nFor low-resolution videos (e.g., 360p or 480p), traditional upscaling algorithms just make the blur bigger. We need the AI to \"hallucinate\" and reconstruct lost details.\n\nThe Model: I use an IMDN (Information Multi-distillation Network) architecture via onnxruntime-web.\n\nThe Execution: The model runs inside a Web Worker. Because the input resolution is small, the tensor allocations (even when batched or padded) easily fit within standard WebGPU limits (usually 128MB max buffer size). The AI does a fantastic job of reconstructing sharp edges, hair, and textures.\n\n**⚡ Pipeline 2: The HD Path (AMD FSR 1.0 via WGSL Shaders)**\n\nWhen a user uploads a 1080p video and wants 4K, running that through an ONNX model in a browser tab takes seconds per frame and usually crashes.\n\nInstead of AI, I implemented AMD's FSR 1.0 (FidelityFX Super Resolution) entirely in custom WGSL Compute Shaders, completely bypassing ONNX.\n\nSince this is pure GPU math running natively via WebGPU, it processes 1080p -> 4K frames in milliseconds. It’s not AI, but at HD resolutions, the pixel density is already high enough that a high-quality spatial upscaler looks nearly identical to AI, but at 1/100th the compute cost.\n\n**🛠️ The Data Flow (Avoiding CPU/GPU ping-pong)**\n\nIn both pipelines, the biggest bottleneck isn't the math; it's moving data between the CPU (JavaScript) and the GPU. I use mediabunny (a wrapper over WebCodecs/FFmpeg) to extract VideoFrame objects. These are passed directly to WebGPU using copyExternalImageToTexture. The processing happens entirely in VRAM, and we only map the final output buffer back to the CPU to paint it onto an OffscreenCanvas before re-encoding it into an MP4 container.\n\n**✅ The Takeaway**\n\nIf you are building client-side AI tools, don't blindly throw Neural Networks at every problem. Fallback mechanisms are your best friend. Using AI where it matters (low-res) and blazing-fast GPU shaders where it doesn't (high-res) is the only way to build a production-ready browser upscaler today.\n\nHas anyone else played around with porting FSR or DLSS-like shaders to WebGPU? Let’s chat in the comments!", "url": "https://wpnews.pro/news/i-built-a-hybrid-ai-spatial-video-upscaler-in-the-browser", "canonical_source": "https://dev.to/hlinhbuilds/i-built-a-hybrid-ai-spatial-video-upscaler-in-the-browser-4o0", "published_at": "2026-09-02 13:01:04+00:00", "updated_at": "2026-09-02 13:25:08.957864+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "ai-infrastructure", "computer-vision"], "entities": ["tapirconvert", "IMDN", "AMD FSR", "WebGPU", "ONNX Runtime", "Real-ESRGAN", "ESPCN", "mediabunny"], "alternates": {"html": "https://wpnews.pro/news/i-built-a-hybrid-ai-spatial-video-upscaler-in-the-browser", "markdown": "https://wpnews.pro/news/i-built-a-hybrid-ai-spatial-video-upscaler-in-the-browser.md", "text": "https://wpnews.pro/news/i-built-a-hybrid-ai-spatial-video-upscaler-in-the-browser.txt", "jsonld": "https://wpnews.pro/news/i-built-a-hybrid-ai-spatial-video-upscaler-in-the-browser.jsonld"}}