Run AI in the Browser: Client-Side Background Removal and Feature Extraction with ONNX Runtime Web

A developer demonstrates how to run AI models directly in the browser using ONNX Runtime Web, eliminating the need for server-side processing. The approach leverages WebAssembly and WebGPU to execute computer vision tasks like background removal and feature extraction client-side, reducing latency and addressing privacy concerns.

For years, heavy computational tasks like computer vision, semantic segmentation, and deep feature extraction were strictly locked behind powerful backend server clusters equipped with expensive GPU arrays. When a user uploaded an image for background removal or semantic profiling, the browser acted merely as a thin presentation layer. It captured pixels, compressed them into JPEGs, and shipped them over HTTP to a remote Python microservice running PyTorch or TensorFlow. Then, it waited for the server to reply. This client-server round-trip introduces severe friction: high latency, massive bandwidth consumption, recurring cloud infrastructure bills, and deep privacy concerns. Routing private user video streams or sensitive enterprise assets through centralized cloud endpoints invites compliance nightmares under frameworks like GDPR and HIPAA. Today, that architecture is obsolete. Thanks to the convergence of advanced hardware acceleration APIs, the maturation of WebAssembly Wasm , and portable machine learning runtimes, we can now execute complex deep learning inference directly inside the browser. The client device is no longer a passive terminal—it is a sovereign edge-computing node. In this guide, we will explore how to build high-performance, client-side computer vision pipelines using ONNX Runtime Web onnxruntime-web , transforming raw HTML5 canvas pixels into real-time segmentation masks and dense vector embeddings entirely in the browser memory space. To appreciate why running transformer-based vision models in the browser is a game-changer, we must look at how client-side intelligence reshapes modern web engineering. When you process images locally, the network bottleneck vanishes. Data never leaves the user’s device. However, bridging the gap between high-level web applications and low-level neural networks requires a robust architectural stack. Client-side machine learning relies on three foundational pillars: onnxruntime-web . Written in C++ and compiled to WebAssembly with WebGL and WebGPU bindings, it serves as the translation layer between TypeScript and browser compute substrates. onnxruntime-web routes tensor math through multi-threaded WebAssembly SIMD CPU cores , WebGL fragment shaders, or native WebGPU compute shaders.When working with feature extraction models such as CLIP vision encoders or lightweight MobileNet backbones , you generate Embedding Vectors . In traditional web development, a hash map or relational database index lets you look up exact keys instantly $O 1 $ complexity . However, if you query a standard database index for a key that is spelled slightly differently or represents a conceptually similar entity, the lookup fails completely. An embedding vector solves this by projecting discrete concepts into a continuous multi-dimensional geometric space. Each dimension in the vector represents a latent feature learned during training. Just as a modern microservices architecture decouples a monolith into independent, specialized services that communicate over a well-defined network mesh, an embedding model decouples raw, unstructured media—pixels, audio waveforms, text documents—into structured, dense numerical coordinates. These coordinates can be compared instantly using mathematical distance metrics like cosine similarity or Euclidean distance, enabling lightning-fast semantic searches directly in client memory. JavaScript is historically single-threaded or operates via message-passing Web Workers and dynamically typed. Deep learning, conversely, requires massive parallel execution of matrix multiplications over contiguous blocks of memory. How does onnxruntime-web bridge this chasm? The runtime employs a multi-tiered execution provider strategy. When an inference session initializes, the runtime inspects the host environment to select the most performant execution provider available: Every modern browser supports WebAssembly, a binary instruction format designed for near-native execution speeds. The Wasm execution provider compiles the core ONNX Runtime C++ engine into a Wasm module. To maximize performance, it leverages SIMD Single Instruction, Multiple Data instructions and multi-threading via Web Workers. While CPU execution is universally supported, it is fundamentally limited by core counts and vector register widths, making it less optimal for dense transformer models. Before WebGPU, WebGL was the primary bridge to hardware acceleration in the browser. Originally designed for 3D graphics, WebGL allows developers to execute custom programs called fragment shaders on the GPU. The ONNX Runtime Web WebGL provider cleverly maps tensor operations onto graphics operations. Multi-dimensional tensors are packed into 2D WebGL textures, and matrix multiplication is executed by rendering a full-screen quad where each pixel output corresponds to a dot product calculated by a fragment shader. While ingenious, WebGL introduces overhead due to texture allocations, state switching, and forcing general-purpose compute into a graphics pipeline. WebGPU represents the modern gold standard for client-side compute. Built from the ground up to expose modern GPU architectures similar to Vulkan, Metal, and DirectX 12 , it provides first-class support for general-purpose GPU GPGPU compute shaders, direct memory management, storage buffers, and compute pipelines. Tensors are stored directly in GPU storage buffers, and compute shaders execute parallel matrix multiplications without the abstraction penalty of rendering fake graphics primitives. This yields inference speeds that approach native desktop application performance, making real-time segmentation of high-definition video streams entirely feasible in the browser. Building a background removal pipeline requires understanding the underlying computer vision models deployed on edge devices such as MediaPipe Selfie Segmentation, RMBG models, or MODNet variants . The pipeline operates across three mathematical stages: Preprocessing , Inference Forward Pass , and Post-Processing . A webcam frame captured via an HTML5 <canvas element yields an ImageData object containing a flat Uint8ClampedArray of RGBA pixels in row-major order, ranging from $0$ to $255$. Neural networks, however, expect multi-dimensional tensors containing floating-point values normalized to a specific range typically $ -1, 1 $ or $ 0, 1 $ with specific channel orderings. Mathematically, if $I {raw}$ is an input pixel value in the range $ 0, 255 $, the preprocessing function applies an affine transformation: $$I {norm} = \frac{\frac{I {raw}}{255.0} - \mu}{\sigma}$$ Furthermore, spatial layouts must be transformed from interleaved channels RGBRGB... to planar format RRR...GGG...BBB... , matching the NCHW Batch, Channels, Height, Width tensor memory layout required by ONNX models. Once the input tensor $X \in \mathbb{R}^{1 \times 3 \times H \times W}$ is passed to the ONNX session, the encoder progressively downsamples spatial dimensions while increasing channel depth, capturing high-level semantic features. The decoder then upsamples these features, combining them with skip-connections to recover spatial precision, ultimately producing a probability map or logit mask $Y \in \mathbb{R}^{1 \times 1 \times H' \times W'}$. The raw output tensor from the model is rarely a crisp binary mask. Due to downsampling and interpolation, edges can be soft or aliased. Post-processing involves: $$C {out} = \alpha \cdot C {foreground} + 1 - \alpha \cdot C {background}$$ Below is a fully self-contained, production-ready TypeScript implementation for a SaaS web application requiring client-side background removal via ONNX Runtime Web. import as ort from 'onnxruntime-web'; / Interface representing the configuration options for the client-side background removal processor. / interface BackgroundRemovalConfig { modelPath: string; executionProvider: 'webgl' | 'wasm' | 'webgpu'; inputWidth: number; inputHeight: number; } / SaaS Feature Processor: Handles client-side background removal using ONNX Runtime Web to offload heavy AI computation from the backend infrastructure. / export class ClientSideBackgroundRemover { private session: ort.InferenceSession | null = null; private config: BackgroundRemovalConfig; private isInitialized: boolean = false; / Initializes the processor with specific runtime configurations. @param {BackgroundRemovalConfig} config - Configuration parameters for the ONNX session. / constructor config: BackgroundRemovalConfig { this.config = config; // Configure ONNX Runtime Web global settings for WASM paths // In a production SaaS, these assets should be served from a CDN or public static folder. ort.env.wasm.wasmPaths = 'https://cdn.jsdelivr.net/npm/onnxruntime-web@latest/dist/'; } / Asynchronously initializes the ONNX Inference Session. This loads the model weights into browser memory. / public async initialize : Promise<void { if this.isInitialized && this.session { return; } try { console.log BackgroundRemover Loading model from ${this.config.modelPath}... ; // Set execution provider priority based on configuration const executionProviders = this.config.executionProvider, 'webgl', 'wasm' ; // Create the inference session with optimized threading settings this.session = await ort.InferenceSession.create this.config.modelPath, { executionProviders: executionProviders, graphOptimizationLevel: 'all', } ; this.isInitialized = true; console.log ' BackgroundRemover Model loaded successfully into client browser.' ; } catch error { console.error ' BackgroundRemover Failed to initialize ONNX session:', error ; throw new Error Initialization failed: ${ error as Error .message} ; } } / Preprocesses an HTMLImageElement or ImageData into a normalized Float32Array tensor matching the input shape requirements of the segmentation model 1x3HxW . @param {HTMLImageElement} image - The source image element provided by the user. @returns {Promise<ort.Tensor } The formatted input tensor. / private async preprocess image: HTMLImageElement : Promise<ort.Tensor { const canvas = document.createElement 'canvas' ; canvas.width = this.config.inputWidth; canvas.height = this.config.inputHeight; const ctx = canvas.getContext '2d' ; if ctx { throw new Error 'Failed to acquire 2D rendering context for image preprocessing.' ; } // Draw and resize image to model input dimensions ctx.drawImage image, 0, 0, this.config.inputWidth, this.config.inputHeight ; const imageData = ctx.getImageData 0, 0, this.config.inputWidth, this.config.inputHeight ; const { data } = imageData; const totalPixels = this.config.inputWidth this.config.inputHeight; // Allocate Float32Array for Planar format Channels-First: 1, 3, H, W const redArray = new Float32Array totalPixels ; const greenArray = new Float32Array totalPixels ; const blueArray = new Float32Array totalPixels ; for let i = 0; i < totalPixels; i++ { const stride = i 4; // Normalize pixel values from 0, 255 to 0.0, 1.0 redArray i = data stride / 255.0; greenArray i = data stride + 1 / 255.0; blueArray i = data stride + 2 / 255.0; } // Concatenate channels into a single contiguous Float32Array const inputData = new Float32Array 3 totalPixels ; inputData.set redArray, 0 ; inputData.set greenArray, totalPixels ; inputData.set blueArray, totalPixels 2 ; // Create the ONNX Tensor with shape 1, 3, Height, Width const tensor = new ort.Tensor 'float32', inputData, 1, 3, this.config.inputHeight, this.config.inputWidth ; return tensor; } / Executes client-side background removal inference on a target image element. @param {HTMLImageElement} sourceImage - The image to process. @returns {Promise<ImageData } The resulting foreground image with a transparent background. / public async removeBackground sourceImage: HTMLImageElement : Promise<ImageData { if this.isInitialized || this.session { throw new Error 'BackgroundRemover has not been initialized. Call initialize first.' ; } // Step 1: Preprocess input image into an ONNX tensor const inputTensor = await this.preprocess sourceImage ; // Extract input name dynamically from model metadata const inputName = this.session.inputNames 0 ; const feeds: Record<string, ort.Tensor = { inputName : inputTensor }; try { // Step 2: Execute model inference forward pass const results = await this.session.run feeds ; const outputName = this.session.outputNames 0 ; const outputTensor = results outputName ; // Step 3: Post-process the output mask and apply alpha blending const processedImageData = this.postprocess sourceImage, outputTensor ; // Clean up tensor memory to prevent V8 memory leaks inputTensor.dispose ; outputTensor.dispose ; return processedImageData; } catch error { inputTensor.dispose ; console.error ' BackgroundRemover Inference execution failed:', error ; throw error; } } / Post-processes the model output mask, resizing it to the original image dimensions and applying an alpha mask to strip the background. / private postprocess sourceImage: HTMLImageElement, maskTensor: ort.Tensor : ImageData { const origWidth = sourceImage.naturalWidth || sourceImage.width; const origHeight = sourceImage.naturalHeight || sourceImage.height; // Render source image to a staging canvas const sourceCanvas = document.createElement 'canvas' ; sourceCanvas.width = origWidth; sourceCanvas.height = origHeight; const sourceCtx = sourceCanvas.getContext '2d' ; if sourceCtx throw new Error 'Could not create source rendering context.' ; sourceCtx.drawImage sourceImage, 0, 0, origWidth, origHeight ; const sourceImageData = sourceCtx.getImageData 0, 0, origWidth, origHeight ; // Create a temporary canvas for scaling the model's output mask const maskCanvas = document.createElement 'canvas' ; maskCanvas.width = this.config.inputWidth; maskCanvas.height = this.config.inputHeight; const maskCtx = maskCanvas.getContext '2d' ; if maskCtx throw new Error 'Could not create mask rendering context.' ; const maskDataFloat = maskTensor.data as Float32Array; const maskImageData = maskCtx.createImageData this.config.inputWidth, this.config.inputHeight ; // Convert raw probability logits into alpha values for let i = 0; i < maskDataFloat.length; i++ { const alphaValue = Math.floor maskDataFloat i 255 ; const stride = i 4; maskImageData.data stride = 255; // R maskImageData.data stride + 1 = 255; // G maskImageData.data stride + 2 = 255; // B maskImageData.data stride + 3 = alphaValue; // Alpha mask } maskCtx.putImageData maskImageData, 0, 0 ; // Scale mask back up to original image dimensions using smooth scaling const finalCanvas = document.createElement 'canvas' ; finalCanvas.width = origWidth; finalCanvas.height = origHeight; const finalCtx = finalCanvas.getContext '2d' ; if finalCtx throw new Error 'Could not create final rendering context.' ; finalCtx.drawImage maskCanvas, 0, 0, origWidth, origHeight ; const expandedMask = finalCtx.getImageData 0, 0, origWidth, origHeight ; // Apply alpha mask to the original image pixels const outputImageData = finalCtx.createImageData origWidth, origHeight ; const srcPixels = sourceImageData.data; const maskPixels = expandedMask.data; const outPixels = outputImageData.data; for let i = 0; i < srcPixels.length; i += 4 { outPixels i = srcPixels i ; // R outPixels i + 1 = srcPixels i + 1 ; // G outPixels i + 2 = srcPixels i + 2 ; // B // Multiply original alpha by normalized mask prediction outPixels i + 3 = srcPixels i + 3 maskPixels i + 3 / 255 ; } return outputImageData; } } Deploying deep learning models to the client side introduces unique engineering constraints that do not exist in server environments: Neural network models require contiguous memory blocks for tensor allocations. In JavaScript, heavy reliance on standard TypedArrays Float32Array can trigger frequent Garbage Collection GC pauses if memory is allocated and discarded inside render loops. High-performance client-side engines must implement strict tensor pooling and memory reuse strategies, explicitly calling .dispose on ONNX tensor handles to prevent memory leaks in the V8 heap. Client-side models must be downloaded by the browser upon initial page load. Quantized transformer models can range from 15 megabytes to over 100 megabytes. Engineering teams must implement intelligent lazy loading strategies, progressive model fetching with visual progress indicators, and browser caching via the Cache API or IndexedDB to ensure subsequent visits load instantly. Not all users possess high-end GPUs. A robust client-side vision engine must implement graceful fallback mechanisms. If WebGPU initialization fails due to outdated graphics drivers, the engine must catch the exception, fall back to WebGL, and if that fails, seamlessly degrade to multi-threaded WebAssembly CPU execution. This ensures application functionality across a diverse spectrum of client devices, from high-end workstations to budget mobile phones. Client-side background removal and feature extraction with ONNX Runtime Web represent the ultimate fusion of systems programming, computer vision mathematics, and modern web browser capabilities. By shifting computational burdens from centralized server clusters to end-user browsers, developers achieve ultra-low latency, enhanced data privacy, and zero server infrastructure costs for real-time media processing pipelines. Through careful management of hardware abstraction layers, rigorous tensor memory lifecycle management, and robust TypeScript architectures, the browser evolves from a simple document viewer into an autonomous edge-computing node capable of real-time visual intelligence. The concepts and code demonstrated here are drawn directly from the comprehensive roadmap laid out in the book Generative Media & Visual Workflow Engines. Node-Based AI Canvases, Real-Time Media Streaming Pipelines, and WebGPU Processing in TypeScript , you can find it here http://tiny.cc/GenerativeMedia . Check also the many other ebooks http://tiny.cc/ProgrammingBooks .