{"slug": "three-things-that-broke-when-i-moved-ai-image-models-into-the-browser", "title": "Three things that broke when I moved AI image models into the browser", "summary": "A developer rebuilt a retired photo-editing site so that object removal (MI-GAN, LaMa), background removal (RMBG-1.4) and 4× upscaling (Real-ESRGAN) all run client-side via ONNX Runtime Web with WebGPU and a WebAssembly fallback, and documented three unexpected failures encountered in the process. BiRefNet-lite exceeded Apple WebGPU's maxStorageBuffersPerShaderStage limit of 10 and exhausted the 4 GB wasm32 heap, LaMa's Fourier convolutions produced wrong values on the WebGPU execution provider (hole mean 254.3 versus 107.3 on WASM), and Real-ESRGAN's fp16 outputs decoded as solid black until the code accepted both Uint16Array and native Float16Array. The developer's stated lessons: benchmark candidate models on the weakest target GPU before comparing quality, and test pictures rather than the absence of errors.", "body_md": "I rebuilt a retired photo-editing site so that every model runs client-side: object removal (MI-GAN, LaMa), background removal (RMBG-1.4) and 4× upscaling (Real-ESRGAN), all through ONNX Runtime Web with WebGPU and a WebAssembly fallback. No uploads, no server, models downloaded once after the user agrees and cached in Cache Storage.\n\nIt works, but three things broke in ways I did not expect. None of them threw an error that pointed at the real cause.\n\nFor background removal, BiRefNet-lite (MIT) beat RMBG-1.4 clearly in my offline comparison on ten images. In the browser on an Apple GPU, the first `session.run()` failed with:\n\n```\nToo many storage buffers in shader. Current: 11, Max is 10\n```\n\nWebGPU on Apple hardware reports `maxStorageBuffersPerShaderStage = 10`, and one of the fused kernels ONNX Runtime generates for this model needs 11. You cannot request more than the adapter offers. Dropping the graph optimisation level did not help, and the WebAssembly backend hit `std::bad_alloc` because the 1024×1024 transformer activations do not fit in a 4 GB wasm32 heap. BEN2 failed the same way.\n\n**Lesson:** benchmark candidate models in the target browser on the weakest GPU you care about before comparing quality. A convolutional model (RMBG-1.4) runs in 0.25 s on WebGPU and ~6 s on WASM; the \"better\" model does not run at all.\n\nLaMa ran on WebGPU without any error. The output tensor had the right shape and values in 0–255. But the inpainted hole was almost pure white:\n\n```\nwebgpu  hole mean=254.3   outside mean=127.0\nwasm    hole mean=107.3   outside mean=127.0\n```\n\nLaMa relies on Fourier convolutions (RFFT/IRFFT), and on the WebGPU execution provider those produced wrong values. My fallback chain (WebGPU → less optimised graph → WASM) only triggers on exceptions, so it never kicked in. LaMa now always runs on WASM, and the end-to-end tests assert on pixel colours of real outputs, not just \"the run finished\".\n\n**Lesson:** test pictures, not the absence of errors.\n\nReal-ESRGAN x4plus is shipped in fp16 with fp16 inputs and outputs. I encoded inputs into a `Uint16Array` and decoded outputs from raw half-float bits. On current Chrome the upscaled image came out solid black. Chrome now has a native `Float16Array`, and ONNX Runtime Web returns fp16 outputs as real numbers when it is available. Decoding 0.5 as if it were a bit pattern gives roughly zero.\n\nThe fix is to accept both:\n\n``` js\nconst out = raw instanceof Uint16Array\n  ? Float32Array.from(raw, halfBitsToNumber)\n  : Float32Array.from(raw as ArrayLike<number>);\n```\n\nThe same model also failed on WebGPU with `Shape mismatch attempting to re-use buffer` until I pinned its symbolic dimensions with `freeDimensionOverrides: {N: 1, H: 192, W: 192}` and fed it fixed-size tiles.\n\n`env.wasm.wasmBinary`.` connect-src 'self'` and no third-party scripts, embedded in the content site by iframe.\nIf you want to poke at it: [https://inpainting.app/](https://inpainting.app/) — the full write-up with examples is on the site.\n\n*This post was written with AI assistance and reviewed by the author.*", "url": "https://wpnews.pro/news/three-things-that-broke-when-i-moved-ai-image-models-into-the-browser", "canonical_source": "https://dev.to/mno_tao_236ab4649edf4cf9f/three-things-that-broke-when-i-moved-ai-image-models-into-the-browser-4ai5", "published_at": "2026-09-28 10:21:19+00:00", "updated_at": "2026-09-28 10:49:43.305111+00:00", "lang": "en", "topics": ["ai-tools", "computer-vision", "ai-infrastructure", "mlops", "developer-tools"], "entities": ["ONNX Runtime Web", "WebGPU", "WebAssembly", "MI-GAN", "LaMa", "RMBG-1.4", "Real-ESRGAN", "BiRefNet-lite"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/three-things-that-broke-when-i-moved-ai-image-models-into-the-browser", "markdown": "https://wpnews.pro/news/three-things-that-broke-when-i-moved-ai-image-models-into-the-browser.md", "text": "https://wpnews.pro/news/three-things-that-broke-when-i-moved-ai-image-models-into-the-browser.txt", "jsonld": "https://wpnews.pro/news/three-things-that-broke-when-i-moved-ai-image-models-into-the-browser.jsonld"}}