Removing a Photo's Background in the Browser, With No Upload: AI Licenses, ONNX Models, and a Frozen Tab

A developer built a 100% client-side background removal tool using Transformers.js and ONNX Runtime Web, avoiding server uploads. The project overcame an AGPL licensing issue, memory crashes from transformer models, and a frozen browser tab by switching to a lightweight CNN and fixing a main-thread blocking bug.

I wanted to add a background-removal tool to my site's image cluster that stayed true to the 100% client-side processing principle I already use for PDFs and image conversions. The path there was anything but linear: a library dropped over a licensing problem, a carefully chosen model that turned out more limited than expected, and a bug that froze the entire page — not just the tool — during computation. Here's the full build, including the parts that didn't work the first time. The initial idea was broad: remove backgrounds, and maybe unwanted objects too. The two tasks have very different difficulty levels. Removing objects requires inpainting — plausibly reconstructing the erased area — which in practice still means heavy generative models, impractical to run client-side with good quality on an average device. Removing a background , on the other hand, is a segmentation problem: separating a subject from its surroundings. That has much lighter models available, runnable entirely via WebAssembly with no server involved at all. So: background removal only, object removal shelved for later. The first library that looked like a perfect fit turned out to be distributed under AGPL , a strong copyleft license. Free to use — but with a real catch for anyone embedding it in a public, closed-source web service: AGPL can require releasing the full source of the project that embeds it, under the same license. "Free for the end user" and "safe to drop into a closed-source commercial product" are two different questions, and it's worth answering the second one before writing integration code, not after deploying it. Before wiring any "free" AI library into a commercial project, check the exact license, not just the price tag. AGPL, GPL, and other strong copyleft licenses are fine for personal or internal tools, risky for a public closed-source product. The fix: switch to Transformers.js — Hugging Face's library for running ML models in the browser on top of ONNX Runtime Web — with a permissively licensed model Apache-2.0 instead of the AGPL wrapper. Same principle an ONNX model pulled once from a CDN, then cached by the browser , clean license. Picking the model taught me a lesson that generalizes to any browser-ML project: architecture matters more than file size . A transformer-based model, even a qualitatively great one with a permissive license, can crash from memory exhaustion during WASM inference on a lot of machines. Transformer attention scales memory with the number of image "patches," and at normal photo resolution the intermediate tensors get huge. A smaller CNN with an architecture built for segmentation runs far more reliably in WASM. | Layer | Choice | |---|---| | Runtime | Transformers.js on top of ONNX Runtime Web WASM | | Default model | Light CNN, Apache-2.0, optimized for people | | Optional model | Heavier transformer, MIT, more general-purpose | | Processing | 100% local, no image upload | | Cache | Browser + service worker, one-time download | The model often left a residual color halo around the cutout, because the output is a soft alpha matte rather than a clean binary mask — background pixels could carry residual alpha values of 5–15% instead of a clean 0. No second model needed. Just post-process the alpha curve: force low values to full transparency, force high values to full opacity, use a smoothstep transition in between so soft edges hair, fine detail survive. On a test image this pushed over 60% of pixels to fully transparent and over 35% to fully opaque, leaving only a small sliver in the mid-range — a clean result. // Simplified alpha cleanup function cleanAlpha a, lowT = 0.08, highT = 0.85 { if a <= lowT return 0; if a = highT return 1; const t = a - lowT / highT - lowT ; return t t 3 - 2 t ; // smoothstep } First real-world tests were disappointing on two fronts: a flat illustration the model only trimmed the margin around the shape, leaving the whole drawing opaque and a dim, cluttered indoor photo the model nearly erased the main subject too . To isolate the cause, I temporarily added a debug panel showing the model's raw mask, before any alpha cleanup, next to the final result. They were nearly identical — so the bug wasn't in my post-processing, it was upstream, in the model itself. Checking the model card confirmed it: the default model was trained specifically on a human segmentation dataset , not a general-purpose one. On a well-lit, close-up portrait, the cutout is genuinely clean, hair included. On anything far from that training domain — illustrations, dark scenes, cluttered framing — quality drops in a predictable, not-a-bug way. A model that's "light and WASM-reliable" is often exactly that becauseit's trained on a narrow domain. Before calling a model "bad," check what it was actually trained on — the mismatch is usually domain, not quality. Instead of replacing the light model with a heavier, more general one for everyone — making every user pay the load-time cost for a case they may never hit — I added a second model an MIT-licensed transformer, better suited to generic scenes available only on request. A "Try the better model" button appears after the first result and reprocesses the same image, without overwriting the existing output until the heavier model succeeds. If it fails from insufficient memory, the first result stays intact and the user gets a clear error instead of a broken tool. Light by default, heavy on explicit request — nobody pays for a download they'll never use. Testing the heavier model surfaced something much worse than mediocre output: during processing, the entire page stopped responding — not just the tool, the top nav links too. The cause: by default, ONNX Runtime Web runs WASM computation on the browser's main thread , the same one handling clicks, scroll, and rendering. During heavy inference that thread stays busy and the whole UI freezes. First attempt at a fix — flipping an internal library option meant to delegate computation to a separate thread — didn't work, likely due to delicate initialization timing. The reliable fix was writing and controlling a real dedicated Web Worker myself: a separate thread that loads the library, downloads the model, and runs the whole inference, talking to the main page only through postMessage input image in, progress and result out . js // worker.js simplified self.onmessage = async e = { const { imageBuffer } = e.data; const { pipeline } = await import 'https://cdn.jsdelivr.net/npm/@huggingface/transformers' ; const remover = await pipeline 'background-removal', MODEL ID, { device: 'wasm' } ; const result = await remover imageBuffer ; self.postMessage { type: 'done', result }, result.buffer ; }; This way the main thread never does heavy computation by construction , not by a configuration flag I was hoping would hold. If client-side AI processing freezes the whole interface, not just the component using it, the top suspect is main-thread computation. A config flag might not be enough — a dedicated, explicitly written Web Worker is the more reliable fix. Even with the freeze fixed, there was a perception problem left: the heavier model takes roughly two minutes on common hardware, and the library doesn't expose real percentage progress during inference — only during the model download. A progress bar sitting "full and still" for two minutes reads as broken , not slow . Final approach: a bar that advances on an estimated curve over ~2.5 minutes, with elapsed seconds shown below, and — if processing runs past that estimate — an automatic switch to an indeterminate animation a continuously scrolling stripe , the universal "still working" signal without faking a percentage the model can't provide. A wider safety timeout unlocks the UI if something really goes wrong, so the user can retry. The most useful lesson isn't about one bug — it's about ordering priorities: If you've hit the AGPL-vs-permissive-license question before, or fought with onnxruntime-web blocking a UI thread, I'd be curious to hear how you solved it. I write about building and maintaining roversia.it — a personal site with 35+ browser tools, PWA games, and small web apps, all vanilla JS, zero monthly cost, and as much as possible zero server-side processing.