{"slug": "i-open-sourced-a-browser-based-ai-background-remover-here-s-the-full", "title": "I Open-Sourced a Browser-Based AI Background Remover — Here's the Full Architecture", "summary": "An open-source, browser-based AI background removal tool that processes images entirely on the client side, ensuring user privacy. The system uses the `@imgly/background-removal` library with an ONNX model and WebAssembly inference, running through five stages from image upload to transparent PNG download. A key feature is the editable grayscale mask, which allows users to manually correct AI imperfections on hair edges or transparent objects.", "body_md": "Most background removal tools work like this: upload your photo to a server, wait for an AI model to process it, download the result. Your image sits on someone else's infrastructure. You hope they delete it.\n\nI built one that works differently. The AI model runs **in your browser tab**. Your image never leaves your device. And I just [open-sourced the core logic](https://github.com/2645149786-dotcom/toolknit/tree/main/open-source/background-remover-standalone) — two files, zero dependencies beyond a CDN import.\n\nHere's how it works under the hood.\n\n## The Pipeline\n\nThe full flow from \"user drops an image\" to \"transparent PNG download\" goes through five stages:\n\n```\nUpload → ONNX Model Load → WebAssembly Inference → Mask Generation → Canvas Compositing\n```\n\nEach stage runs entirely client-side. Let me walk through them.\n\n## Stage 1: Loading the AI Model in the Browser\n\nThe backbone is [ @imgly/background-removal](https://img.ly/blog/background-removal-js/), an open-source library that bundles an ONNX segmentation model with ONNX Runtime Web (WebAssembly backend).\n\n``` js\nconst LIB_CDN = 'https://cdn.jsdelivr.net/npm/@imgly/background-removal@1.5.5';\n\nasync function loadLibrary() {\n  const module = await import(LIB_CDN + '/+esm');\n  removeBackgroundFn = module.removeBackground;\n}\n```\n\nThe first call downloads ~40MB of model weights. That sounds heavy, but:\n\n- The browser caches it automatically\n- Subsequent uses load instantly from cache\n- No server round-trip on any future use\n\nThis is the same trade-off FFmpeg.wasm makes — big initial download, but then your browser becomes a local processing powerhouse.\n\n## Stage 2: Running AI Inference Locally\n\nOnce the model is loaded, inference is straightforward:\n\n``` js\nconst imageBlob = await new Promise(r => canvas.toBlob(r, 'image/png'));\n\nconst resultBlob = await removeBackgroundFn(imageBlob, {\n  model: 'medium',\n  output: { format: 'image/png' },\n  progress: (key, current, total) => {\n    // Update loading UI\n  }\n});\n```\n\nWhat's happening behind the scenes:\n\n- The library resizes your image to the model's input dimensions\n- Pixel data is converted to a tensor\n- ONNX Runtime Web runs the segmentation model via WebAssembly\n- The output tensor (a per-pixel foreground probability map) is converted back to an image with transparent background\n\nThe `medium`\n\nmodel balances quality and speed. On a decent laptop, inference takes 2-5 seconds for a typical photo. On a phone, maybe 8-15 seconds. Acceptable for a free, private tool.\n\n## Stage 3: Building the Editable Mask\n\nHere's where it gets interesting. The AI output isn't final — it's a starting point. I extract the alpha channel from the AI result and build an editable grayscale mask:\n\n``` js\nasync function buildMaskFromResult() {\n  const w = originalImage.naturalWidth;\n  const h = originalImage.naturalHeight;\n\n  // Draw AI result to a temporary canvas\n  const resultCanvas = document.createElement('canvas');\n  resultCanvas.width = w;\n  resultCanvas.height = h;\n  const rCtx = resultCanvas.getContext('2d');\n  rCtx.drawImage(resultImg, 0, 0);\n  const resultData = rCtx.getImageData(0, 0, w, h);\n\n  // Extract alpha channel → grayscale mask\n  // White = foreground (keep), Black = background (remove)\n  maskCanvas = document.createElement('canvas');\n  maskCanvas.width = w;\n  maskCanvas.height = h;\n  maskCtx = maskCanvas.getContext('2d');\n  const maskData = maskCtx.createImageData(w, h);\n\n  for (let i = 0; i < resultData.data.length; i += 4) {\n    const alpha = resultData.data[i + 3];\n    maskData.data[i] = alpha;     // R\n    maskData.data[i + 1] = alpha; // G\n    maskData.data[i + 2] = alpha; // B\n    maskData.data[i + 3] = 255;   // A (mask itself is always opaque)\n  }\n  maskCtx.putImageData(maskData, 0, 0);\n}\n```\n\n**Why a separate mask canvas?**\n\nBecause users need to fix the AI's mistakes. Hair edges, transparent objects, similar-colored backgrounds — no AI gets these perfect 100% of the time. The mask canvas becomes a paintable surface.\n\n## Stage 4: Manual Refinement with Brush & Eraser\n\nThis is the feature that separates a toy demo from a usable tool. Users can:\n\n-\n**Brush**(paint white on mask) → restore foreground areas the AI removed -\n**Eraser**(paint black on mask) → remove background areas the AI missed\n\n``` js\nfunction paintOnMask(e) {\n  const rect = editCanvas.getBoundingClientRect();\n  const x = (e.clientX - rect.left) / rect.width * maskCanvas.width;\n  const y = (e.clientY - rect.top) / rect.height * maskCanvas.height;\n\n  const brushSize = parseInt(brushSizeEl.value);\n  const softness = parseInt(brushSoftEl.value) / 100;\n\n  maskCtx.lineCap = 'round';\n  maskCtx.lineWidth = brushSize;\n\n  // Softness = CSS filter blur on the mask canvas context\n  if (softness > 0) {\n    maskCtx.filter = `blur(${Math.round(brushSize * softness * 0.3)}px)`;\n  }\n\n  if (currentTool === 'brush') {\n    maskCtx.globalCompositeOperation = 'lighter';\n    maskCtx.strokeStyle = '#ffffff';\n  } else {\n    maskCtx.globalCompositeOperation = 'source-over';\n    maskCtx.strokeStyle = '#000000';\n  }\n\n  maskCtx.beginPath();\n  maskCtx.moveTo(lastX, lastY);\n  maskCtx.lineTo(x, y);\n  maskCtx.stroke();\n}\n```\n\n**Key details:**\n\n-\n**Coordinate mapping**: The edit canvas is CSS-scaled to fit the viewport, but the mask operates at full image resolution. Every mouse position gets mapped from display coordinates to mask coordinates. -\n**Edge softness**: Uses Canvas 2D`filter: blur()`\n\non the stroke — this creates feathered edges instead of hard cuts. -\n**Undo stack**: Each mousedown saves a full`ImageData`\n\nsnapshot of the mask. Up to 20 undo levels.\n\nThe brush cursor is a `position: fixed`\n\ndiv that follows the mouse, sized to match the display-scaled brush diameter. The actual canvas cursor is set to `none`\n\n.\n\n## Stage 5: Compositing the Final Output\n\nTo generate the downloadable PNG, the mask is applied to the original image:\n\n``` js\nfunction applyMaskToOriginal() {\n  const origData = origCtx.getImageData(0, 0, w, h);\n  const mData = maskCtx.getImageData(0, 0, w, h);\n  const outData = oCtx.createImageData(w, h);\n\n  for (let i = 0; i < origData.data.length; i += 4) {\n    outData.data[i] = origData.data[i];       // R — original\n    outData.data[i + 1] = origData.data[i + 1]; // G — original\n    outData.data[i + 2] = origData.data[i + 2]; // B — original\n    outData.data[i + 3] = mData.data[i];       // A — from mask R channel\n  }\n\n  oCtx.putImageData(outData, 0, 0);\n  return outCanvas;\n}\n```\n\nThe mask's R channel (which equals G and B since it's grayscale) becomes the alpha channel of the output. White mask pixels → fully opaque. Black → fully transparent. Gray → semi-transparent (useful for hair and soft edges).\n\n## The Refine Mode Overlay\n\nIn refine mode, users see the original image with a semi-transparent red overlay on removed areas:\n\n```\nfunction renderMaskOverlay() {\n  editCtx.drawImage(maskCanvas, 0, 0, dw, dh);\n  const overlayData = editCtx.getImageData(0, 0, dw, dh);\n\n  for (let i = 0; i < overlayData.data.length; i += 4) {\n    const maskVal = overlayData.data[i];\n    if (maskVal < 128) {\n      // Removed area → semi-transparent red\n      overlayData.data[i] = 220;     // R\n      overlayData.data[i + 1] = 50;  // G\n      overlayData.data[i + 2] = 50;  // B\n      overlayData.data[i + 3] = 120; // A\n    } else {\n      // Kept area → fully transparent (show original underneath)\n      overlayData.data[i + 3] = 0;\n    }\n  }\n  editCtx.putImageData(overlayData, 0, 0);\n}\n```\n\nThis gives immediate visual feedback — you can see exactly what the AI removed and paint corrections in real time.\n\n## Performance Considerations\n\n-\n**Memory**: Three full-resolution canvases live in memory (original, mask, output). For a 4000×3000 photo, that's ~144MB of pixel data. Mobile devices with <4GB RAM may struggle. -\n**Real-time rendering**: Every brush stroke triggers`renderPreview()`\n\nvia`requestAnimationFrame`\n\n. This redraws the preview canvas + overlay from the mask. On large images, there's a noticeable lag. -\n**Touch support**: Full touch event handling with`passive: false`\n\nto prevent scroll interference.\n\n## What I Stripped for the Open-Source Version\n\nThe production version on [ToolKnit](https://toolknit.com/tools/background-remover.html) includes:\n\n- Daily usage limits (fair-use throttling)\n- Analytics tracking\n- Self-hosted model weights (faster loading from our CDN)\n- Sound effects on completion\n- Site navigation and SEO shell\n\nThe [open-source version](https://github.com/2645149786-dotcom/toolknit/tree/main/open-source/background-remover-standalone) strips all of that down to two files:\n\n-\n`index.html`\n\n— standalone UI (~250 lines) -\n`app.js`\n\n— core logic (~380 lines)\n\nYou can clone it, run `npx serve .`\n\n, and have a working background remover in 30 seconds.\n\n## What's Next\n\nSome ideas for anyone who wants to fork and extend:\n\n-\n**Background replacement**— solid color or custom image behind the subject -\n**Batch processing**— drop multiple images, process all sequentially -\n**WebGPU acceleration**— ONNX Runtime Web supports WebGPU; inference could be 3-5x faster -\n**Edge feathering controls**— post-process the mask with adjustable blur radius -\n**Before/after slider**— drag to compare original and result\n\n## Try It\n\n-\n**Live tool**:[toolknit.com/tools/background-remover.html](https://toolknit.com/tools/background-remover.html) -\n**Open source**:[github.com/2645149786-dotcom/toolknit](https://github.com/2645149786-dotcom/toolknit/tree/main/open-source/background-remover-standalone) -\n**All 61 tools**:[toolknit.com](https://toolknit.com)\n\nIf you've ever needed to remove a background without uploading your photo to a random website — this is it. Clone it, use it, break it, improve it.\n\n*Built by Zihang Dong. Building browser-first tools at ToolKnit.*", "url": "https://wpnews.pro/news/i-open-sourced-a-browser-based-ai-background-remover-here-s-the-full", "canonical_source": "https://dev.to/dngzihng114379/i-open-sourced-a-browser-based-ai-background-remover-heres-the-full-architecture-1olb", "published_at": "2026-05-20 03:30:59+00:00", "updated_at": "2026-05-20 04:04:01.962051+00:00", "lang": "en", "topics": ["open-source", "artificial-intelligence", "developer-tools", "machine-learning"], "entities": ["@imgly/background-removal", "ONNX Runtime Web", "FFmpeg.wasm"], "alternates": {"html": "https://wpnews.pro/news/i-open-sourced-a-browser-based-ai-background-remover-here-s-the-full", "markdown": "https://wpnews.pro/news/i-open-sourced-a-browser-based-ai-background-remover-here-s-the-full.md", "text": "https://wpnews.pro/news/i-open-sourced-a-browser-based-ai-background-remover-here-s-the-full.txt", "jsonld": "https://wpnews.pro/news/i-open-sourced-a-browser-based-ai-background-remover-here-s-the-full.jsonld"}}