Browser-Based AI: Lessons from Shipping Three Local Tools

The process revealed several critical failure points that aren't documented in standard tutorials—specifically regarding model integrity and runtime bugs.

Detecting Model Corruption via Physics Violations #

My stem splitter (vocal/instrumental separation) uses Meta's HTDemucs. I initially used a third-party ONNX export and wired it up via onnxruntime-web

, but the output was robotic garbage with vocals bleeding into the instrumental track.

I spent hours chasing the wrong ghosts. I suspected my FP16 conversion for smaller download sizes, but comparing FP16 and FP32 natively showed they were numerically identical to four significant figures. I checked the sample rate (44.1kHz stereo), and the logs confirmed the data reaching the model was correct.

To isolate the issue, I reimplemented the _spec()

and _ispec()

(STFT and inverse STFT) from the PyTorch source in JavaScript. I compared the outputs against the original PyTorch implementation using identical inputs. Every probed value matched to eight decimal places. My pipeline was perfect; the model was the problem.

I ran a full-band song through both the official PyTorch checkpoint and my ONNX model. The results were physically impossible:

Vocals @ 20–60Hz: Reference (PyTorch) 4.8% of mix vs. ONNX model 69.3%Vocals @ 2–6kHz: Reference (PyTorch) 0.6% of mix vs. ONNX model 75.4%Bass @ sub-bass: Reference (PyTorch) 0% vs. ONNX model 145.2%Other @ sub-bass: Reference (PyTorch) 0% vs. ONNX model 174.7%

You cannot extract more energy from a signal than existed in the original mix. Seeing values at 145% and 174% magnitude was the smoking gun: the model file itself was defective.

The fix was to stop relying on third-party conversions and export my own ONNX from the official pretrained checkpoint. This process is notoriously finicky due to custom STFT op handling. After validating the new export, I hit a 0.987/0.999 correlation with the reference.

Pro Tip: When a model produces "bad" results, don't just rely on subjective quality. Look for a metric that is physically impossible. It's a much faster signal than "it sounds off."

The ONNX Runtime Graph Optimizer Trap #

For speech-to-text, I used Whisper via transformers.js

. I hit a wall during session creation with a cryptic error:

TransposeDQWeightsForMatMulNBits Missing required scale ... model.decoder.embed_tokens.weight_transposed_DequantizeLinear

At first glance, this looked like a quantization error. MatMulNBits

is a 4-bit operator, but I was using a standard uint8 build. I searched the binary for MatMulNBits

strings—there were none.

The culprit wasn't the model file, but the ONNX Runtime graph optimizer. It was creating the 4-bit op on the fly during session creation because the model used tied embeddings (where the encoder and decoder share the same weight matrix). The optimizer tried to optimize the transpose operation for these tied weights but failed to provide the required scale parameter.

To fix this, I had to explicitly disable the specific optimization that was triggering the bug. In my onnxruntime-web

configuration, I passed the following session options:

const sessionOptions = {
  graphOptimizationLevel: 'basic', // Avoid 'extended' or 'all' to skip the buggy optimizer pass
  executionMode: 'sequential',
};

const session = await ort.InferenceSession.create('/path/to/whisper.onnx', sessionOptions);

By dropping the optimization level from all

to basic

, the runtime stopped attempting the faulty weight transposition, and the model loaded instantly.

Memory Ceilings and Large File Crashes #

The final hurdle was the browser's memory limit. While small files worked perfectly, long audio files triggered browser crashes or "Out of Memory" (OOM) errors. This is because onnxruntime-web

often allocates large contiguous buffers for tensors.

I discovered that the memory ceiling isn't just about the total RAM available, but about how the browser manages the WASM heap. For long-form audio, the STFT transforms create massive intermediate tensors that spike memory usage far beyond the final output size.

To solve this, I implemented a sliding window processing strategy:

1. Split the audio into 30-second chunks.

2. Process each chunk through the model independently.

3. Apply a small cross-fade (approx 100ms) at the boundaries to prevent audible clicking.

4. Reassemble the processed chunks in a Float32Array

before final export.

This kept the peak memory usage constant regardless of the file length, turning a tool that crashed at 5 minutes into one that could handle hour-long recordings.

Next JWT Security: 6 Common Failures in Open Source →