{"slug": "browser-based-ai-lessons-from-shipping-three-local-tools", "title": "Browser-Based AI: Lessons from Shipping Three Local Tools", "summary": "A developer shipping three browser-based AI tools—a stem splitter using Meta's HTDemucs, a Whisper speech-to-text model, and a third unnamed tool—found that model corruption, ONNX Runtime graph optimizer bugs, and browser memory ceilings caused failures that standard tutorials do not document. The stem splitter's third-party ONNX export produced physically impossible outputs (e.g., 174.7% energy extraction), forcing the developer to export their own ONNX from the official checkpoint. The Whisper model failed due to the ONNX Runtime graph optimizer creating a 4-bit op without required parameters, fixed by lowering the optimization level to 'basic'. Large audio files triggered browser crashes from memory limits.", "body_md": "# Browser-Based AI: Lessons from Shipping Three Local Tools\n\nThe process revealed several critical failure points that aren't documented in standard tutorials—specifically regarding model integrity and runtime bugs.\n\n## Detecting Model Corruption via Physics Violations\n\nMy stem splitter (vocal/instrumental separation) uses Meta's HTDemucs. I initially used a third-party ONNX export and wired it up via `onnxruntime-web`\n\n, but the output was robotic garbage with vocals bleeding into the instrumental track.\n\nI spent hours chasing the wrong ghosts. I suspected my FP16 conversion for smaller download sizes, but comparing FP16 and FP32 natively showed they were numerically identical to four significant figures. I checked the sample rate (44.1kHz stereo), and the logs confirmed the data reaching the model was correct.\n\nTo isolate the issue, I reimplemented the `_spec()`\n\nand `_ispec()`\n\n(STFT and inverse STFT) from the PyTorch source in JavaScript. I compared the outputs against the original PyTorch implementation using identical inputs. Every probed value matched to eight decimal places. My pipeline was perfect; the model was the problem.\n\nI ran a full-band song through both the official PyTorch checkpoint and my ONNX model. The results were physically impossible:\n\n**Vocals @ 20–60Hz:** Reference (PyTorch) 4.8% of mix vs. ONNX model 69.3%**Vocals @ 2–6kHz:** Reference (PyTorch) 0.6% of mix vs. ONNX model 75.4%**Bass @ sub-bass:** Reference (PyTorch) 0% vs. ONNX model 145.2%**Other @ sub-bass:** Reference (PyTorch) 0% vs. ONNX model 174.7%\n\nYou cannot extract more energy from a signal than existed in the original mix. Seeing values at 145% and 174% magnitude was the smoking gun: the model file itself was defective.\n\nThe fix was to stop relying on third-party conversions and export my own ONNX from the official pretrained checkpoint. This process is notoriously finicky due to custom STFT op handling. After validating the new export, I hit a 0.987/0.999 correlation with the reference.\n\n**Pro Tip:** When a model produces \"bad\" results, don't just rely on subjective quality. Look for a metric that is physically impossible. It's a much faster signal than \"it sounds off.\"\n\n## The ONNX Runtime Graph Optimizer Trap\n\nFor speech-to-text, I used Whisper via `transformers.js`\n\n. I hit a wall during session creation with a cryptic error:\n\n`TransposeDQWeightsForMatMulNBits Missing required scale ... model.decoder.embed_tokens.weight_transposed_DequantizeLinear`\n\nAt first glance, this looked like a quantization error. `MatMulNBits`\n\nis a 4-bit operator, but I was using a standard uint8 build. I searched the binary for `MatMulNBits`\n\nstrings—there were none.\n\nThe culprit wasn't the model file, but the ONNX Runtime graph optimizer. It was creating the 4-bit op on the fly during session creation because the model used tied embeddings (where the encoder and decoder share the same weight matrix). The optimizer tried to optimize the transpose operation for these tied weights but failed to provide the required scale parameter.\n\nTo fix this, I had to explicitly disable the specific optimization that was triggering the bug. In my `onnxruntime-web`\n\nconfiguration, I passed the following session options:\n\n``` js\nconst sessionOptions = {\n  graphOptimizationLevel: 'basic', // Avoid 'extended' or 'all' to skip the buggy optimizer pass\n  executionMode: 'sequential',\n};\n\nconst session = await ort.InferenceSession.create('/path/to/whisper.onnx', sessionOptions);\n```\n\nBy dropping the optimization level from `all`\n\nto `basic`\n\n, the runtime stopped attempting the faulty weight transposition, and the model loaded instantly.\n\n## Memory Ceilings and Large File Crashes\n\nThe final hurdle was the browser's memory limit. While small files worked perfectly, long audio files triggered browser crashes or \"Out of Memory\" (OOM) errors. This is because `onnxruntime-web`\n\noften allocates large contiguous buffers for tensors.\n\nI discovered that the memory ceiling isn't just about the total RAM available, but about how the browser manages the WASM heap. For long-form audio, the STFT transforms create massive intermediate tensors that spike memory usage far beyond the final output size.\n\nTo solve this, I implemented a sliding window processing strategy:\n\n1. Split the audio into 30-second chunks.\n\n2. Process each chunk through the model independently.\n\n3. Apply a small cross-fade (approx 100ms) at the boundaries to prevent audible clicking.\n\n4. Reassemble the processed chunks in a `Float32Array`\n\nbefore final export.\n\nThis kept the peak memory usage constant regardless of the file length, turning a tool that crashed at 5 minutes into one that could handle hour-long recordings.\n\n[Next JWT Security: 6 Common Failures in Open Source →](/en/threads/2951/)", "url": "https://wpnews.pro/news/browser-based-ai-lessons-from-shipping-three-local-tools", "canonical_source": "https://promptcube3.com/en/threads/2953/", "published_at": "2026-07-24 23:46:48+00:00", "updated_at": "2026-07-25 00:07:12.883999+00:00", "lang": "en", "topics": ["ai-tools", "ai-infrastructure", "developer-tools"], "entities": ["Meta", "HTDemucs", "ONNX Runtime", "Whisper", "transformers.js", "onnxruntime-web"], "alternates": {"html": "https://wpnews.pro/news/browser-based-ai-lessons-from-shipping-three-local-tools", "markdown": "https://wpnews.pro/news/browser-based-ai-lessons-from-shipping-three-local-tools.md", "text": "https://wpnews.pro/news/browser-based-ai-lessons-from-shipping-three-local-tools.txt", "jsonld": "https://wpnews.pro/news/browser-based-ai-lessons-from-shipping-three-local-tools.jsonld"}}