FP16 Quantization Quality Loss: We Blamed the Architecture, the Bug Was One Overflowed Tensor

A developer traced a severe fp16 quantization quality loss in a mask-gated inpainting model to sixteen BatchNorm variance tensors that exceeded fp16's maximum representable value and were silently clamped by the converter, causing those layers to normalize by the wrong divisor. The earlier explanation blaming mask-gated architecture was wrong: tests rounding only weight mantissas showed the model is precision-indifferent, losing just 0.004 dB at fp16's 10-bit mantissa versus fp32. The two-line algebraic fix restores fp32-matching quality while cutting download size by 52%.

Our fp16 quantization post-mortem was wrong in the expensive direction: it told anyone with a mask-gated model to give up on half-precision, and that verdict would have kept a 52%-smaller download off the internet. The real cause was sixteen BatchNorm variance tensors larger than fp16's maximum representable value, silently clamped by the converter, leaving sixteen layers normalizing their output by the wrong divisor. The fix is two lines of algebra. The reward is a model that matches fp32 and downloads 52% smaller. This is a correction post, so it comes with receipts: the lab notes for this round are checked in next to the code, every figure below is reproducible from them on a CPU-only ONNX Runtime install, and the whole investigation fits in a day or two of work. Part 3 of this series https://clearpix.org/blog/fp16-onnx-webgpu-browser-inference ends with a case study called "When fp16 quantization quality loss is real: the LaMa failure". The measurements in it are real, and the failure is reproducible: rebuilding that conversion from the same weights today, the unfixed fp16 model averages 15.8 dB against fp32's 29.7 dB across our 14 acceptance cases, and its worst case scores 11.1 dB — the watermark surviving, exactly as described. Part 5 https://clearpix.org/blog/running-lama-realesrgan-in-the-browser repeats it as "its catastrophic mirror image when we tried it on LaMa". Part 1's series index https://clearpix.org/blog/private-in-browser-watermark-removal-architecture summarises the lesson as "mask-gated architectures and precision sensitivity". We offered a mechanism too, and it was a good story. LaMa is mask-gated: the mask doesn't just mark where to work, it gates the network's behavior. A mask-gated inpainter therefore has a decision boundary inside it — which regions are trustworthy context, which must be regenerated — and half-precision noise pushes the model across that boundary. The gating fails softly, the model starts copying the watermarked pixels through instead of repairing them, and the output looks plausible except the watermark is still there. That story explained the symptom exactly. It was consistent with everything else we'd learned browser inference is bandwidth-bound, so moving half the bytes is free money . And it produced a tidy rule that we then repeated to readers: whether quantization is safe is a property of the model's architecture, not of your tooling The rule is wrong. Not "wrong with exceptions" — wrong as a predictor, and expensive: it would have kept a quality-identical model that is 48% smaller out of the product forever, and it tells every reader with an attention head, a gating mask, or a long residual chain to stop investigating. Part 8 of this series https://clearpix.org/blog/watermark-removal-artifacts-blend-not-model was about a quality bug that turned out to be ours, not the model's. This is the same shape of mistake, one layer lower in the stack: our conversion tooling, not the model's architecture. "FP16" bundles two independent changes: mantissa precision how finely you can split a value and numeric range the largest magnitude you can hold at all . If the failure were precision sensitivity, then rounding the mantissa while keeping fp32's range would degrade quality in proportion. So we built models that keep the graph and all arithmetic in fp32, and only round the weights' mantissas: | Weights rounded to | Region PSNR 5 cases | Output fidelity vs fp32 | |---|---|---| | fp32 baseline | 28.88 | — | | 7-bit mantissa bf16 | 28.83 | 66.9–76.9 dB | | 10-bit mantissa = fp16 | 28.88 | 74.5–96.6 dB | | 13-bit mantissa | 28.88 | 95.9–109.0 dB | fp16 has a 10-bit mantissa. The 10-bit row is 0.004 dB off the fp32 baseline, and even throwing away 16 of the 23 mantissa bits 7-bit, coarser than bf16 costs 0.054 dB. LaMa is not precision-sensitive — it is precision-indifferent. Whatever fp16 was doing to it, it wasn't the mantissa. That left range, and range has a hard, checkable boundary: fp16's largest finite value is 65,504. So we dumped every fp32 initializer in the graph and sorted by magnitude. The graph is 17,480 nodes with 606 float32 initializer tensors, 51M parameters, 204 MB of weights — 197 MB of it in the FFT path 17,022 of those nodes are Fourier convolutions, 97% . Run the check below over the 606 tensors and sixteen come back above 65,504 , and every single one is a BatchNorm running var : 484322.56 model.generator.model.5.conv1.bn l.running var 440853.66 model.generator.model.25.running var 240497.69 model.generator.model.22.conv1.bn l.running var 218544.31 model.generator.model.11.conv1.bn l.running var 174809.92 model.generator.model.6.conv1.bn l.running var 65564.05 model.generator.model.10.conv1.bn l.running var ... 16 of 606 tensors, every one of them a running var Note the last row: the marginal tensor clears the ceiling by 0.09%, and it gets clamped to exactly the same 65,504 as the one that clears it by 7×. That's the whole hazard — a clamped value sits exactly at fp16's ceiling, which is a value fp16 can barely hold, so the next fp16 pass over that tensor takes it to infinity. This is a five-line check, and it is the first thing to run before believing any "this model can't be quantized" conclusion: python import numpy as np, onnx from onnx import numpy helper m = onnx.load "lama fp32.onnx", load external data=False worst = sorted float np.abs numpy helper.to array t .max , t.name for t in m.graph.initializer if t.data type == 1 , 1 = float32 reverse=True, :16 for value, name in worst: print f"{value: 14.2f} {name}" Sixteen tensors. Not sixteen layers of masking logic — sixteen numbers. A BatchNorm layer computes y = x − μ / sqrt var + ε · γ + β . running var is a scale quantity measured over the layer's input during training. If a layer's inputs live at large magnitudes, its variance lives at the square of those magnitudes, and 484,322 is entirely legitimate in fp32: sqrt 484322 ≈ 696 . Clamp that variance to fp16's ceiling of 65,504 and the divisor becomes sqrt 65504 ≈ 256 . The layer now normalizes by 256 instead of 696 — its output is 2.7× too large — and the convolution that consumes it was trained against the correct scale. Sixteen of those, in series, and the network's internal magnitudes are destroyed while the graph still runs and still produces plausible-looking images. That last part is the trap. There is no NaN, no warning, no crash: just a model that quietly keeps the watermark. The conversion never told us anything, and this is the part worth internalising. onnxruntime.transformers.float16.convert float to float16 clamps to max finite val=65504 ; onnxconverter common.float16 defaults to 10000 48× more aggressive . Both clamp silently — that's what "clamp" means: the fp16 model has a finite, wrong number in it, and nothing anywhere prints a line. You can watch the corruption happen tensor by tensor, which is how we finally believed it: | Stage | model.5.conv1.bn l.running var | |---|---| | fp32 source | 484322.5625 | | after keep io types fp16 conversion | 65504.0 clamped to the fp16 ceiling | | after a further fp16 storage pass | inf | | after our fix see below | 1892.0 fits comfortably in fp16 | Note the third row: the clamped value is already at the ceiling, so a second conversion overflows it to infinity. Whether your broken model lands on 65504 or inf depends on which converter pass touched it last — which is also why the failure looked so erratic in our old notes. We tried the blunt version first: keep the sixteen offending BatchNorm nodes in fp32 and convert everything else. That works, costs 34 KB of fp32 parameters, and produced identical quality — but it leaves a graph with fp32 islands and sixteen permanent assertions about which layers are dangerous. The better fix is to notice that a BatchNorm's variance and its scale are only meaningful relative to each other : y = x − μ / sqrt var + ε · γ + β ≡ x − μ / sqrt var/K² + ε/K² · γ/K + β So pick a power-of-two K per layer, divide the variance by K² , divide the scale by K , and divide epsilon by K² — the layer is mathematically identical, and now var/K² fits in fp16. With K = 2^ceil log2 max|var| / 40000 , that's a twenty-line script touching 19 of the 75 BatchNorm layers: k = float 2 np.ceil np.log2 max abs var / 40000.0 K = 16 for the worst layer var init.CopyFrom numpy helper.from array var / k k .astype np.float32 , var init.name scale init.CopyFrom numpy helper.from array scale / k .astype np.float32 , scale init.name and the epsilon attribute: a.f = a.f / k k The result is worth stating precisely, because it kills the "mixed precision is inherently fuzzy" intuition: the converted model has 606 fp16 initializers and zero fp32 residue . It is 100% fp16, every tensor, with no casts added around the BatchNorms. And the arithmetic is exact — powers of two, so nothing is lost to rounding. Three independent measurements on the same converted weights, plus a control and a harness sanity check: | Check | fp32 | fp16 fixed | |---|---|---| | 14-case masked-region PSNR | 29.74 dB | 29.73 dB | | Per-case ghost-score drift mean / max | — | 0.0014 / 0.005 | | Browser runtime ORT Web WASM , same tile, vs fp32 reference | — | 66.5 dB | | Unfixed fp16, same test control | — | 22.6 dB | | 16 real photo regions, production-detected boxes | — | 59.9 dB mean, 40.3 dB min | The browser row is the one that matters for shipping: that's the exact runtime our production HQ tier uses, and 66.5 dB means the two models disagree at the level of sub-grey-step rounding. The control row is what the old mechanism should have looked like if it were real precision damage — 22.6 dB, i.e. visibly different. Sanity check on the harness: browser fp32 vs a CPU fp32 reference on the same tile is 156 dB, so the comparison chain isn't the limiting factor. The size win is the point of the exercise: | | Raw | brotli q11 what users download | |---|---|---| | fp32 | 198.4 MiB | 179.2 MiB | | fp16 fixed | 102.8 MiB | 86.3 MiB −52% | The brotli pipeline that makes a 198 MiB weight file a 179 MiB download in the first place is part 9's story https://clearpix.org/blog/brotli-wasm-models-r2-delivery — worth saying here because the win compounds: same compressor, half the bytes. And now the honest part, which is the real lesson for anyone shipping fp16 in a browser: fp16 made our inpainting slower, not faster. 11.4 s → 17.4 s per tile on the WASM execution provider, 2.7 s → 9.4 s on CPU. The WASM EP has no native fp16 convolution kernels, so it inserts casts and computes in fp32 anyway, paying the copy tax for nothing. Part 3's 3.5× speedup came from WebGPU, which has real fp16 ALUs; on the WebGPU-disabled HQ path, half-precision buys bytes and nothing else. If a future ORT Web release ships fp16 kernels on WASM — or the JSEP gap that keeps HQ off WebGPU gets closed upstream — this model collects the speed win too. The fix is the precondition for that, not the payoff. The old post also reported int8 failing 24.5 → 15.6 dB on our dark-text case and concluded that quantization is architecture-hostile in general. We re-ran both int8 recipes on the current five-case subset our case definitions have drifted since 2026-10, so compare within a run, not across posts : | Variant | Region PSNR | vs fp32 | |---|---|---| | fp32 | 28.88 | — | | int8 dynamic weights only, per-channel | 17.82 | −11.1 dB | | int8 QDQ activations too , all 75 BatchNorms excluded | 19.11 | −9.8 dB | So int8 is still dead for this model — the conclusion stands. But the mechanism is the opposite of the one we published: this isn't a weight-precision problem, and it isn't the BatchNorm overflow either. Explicitly excluding every BatchNorm from quantization doesn't rescue it, because the inputs to those 75 layers still arrive through DequantizeLinear . Counting the graph: 432 Einsum / MatMul nodes with 648 inputs wrapped in quantize/dequantize pairs. The FFT path is expanded into spectral matrix products during export, and that's the part that cannot survive 8-bit activations. Different failure, different size of blast radius, and it stays on the do-not-retry list. Our old order was architecture → bit width → runtime. The new one starts one step earlier, because a conversion artifact can impersonate an architectural verdict: max |value| for every initializer, compare against the target dtype's ceiling, and check the subnormal floor while you're there. In our model 16 of 606 tensors overflowed fp16 0.087% of parameters sat below the subnormal floor — harmless, but the same scan finds both . This costs minutes and can invalidate a conclusion you'd otherwise keep for months. Two corrections to our own record, then: the LaMa fp16 entry comes off the do-not-retry list and onto the opt-in path fp16 verified against fp32 on every check above, fp32 spec retained as the fallback , the int8 entry stays, and the sentence in parts 1 and 5 that makes quantization safety a property of architecture gets retired in favour of something less quotable and more true: a quantization failure is a measurement, and most measurements have a checkable cause. Everything here runs client-side with no uploads. The fp32 model that this post improves powers the free watermark remover https://clearpix.org/remove-watermark , and the inpainting engine is the same one behind the video watermark remover https://clearpix.org/remove-video-watermark . The fp16 path that started all of this is live in our image upscaler https://clearpix.org/upscale-image , and the quality dashboard https://clearpix.org/quality publishes the bench numbers these gates are built on. Part 13 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org https://clearpix.org .