{"slug": "one-linear-layer-broke-my-int8-encoder", "title": "One Linear layer broke my INT8 encoder", "summary": "Dynamic INT8 quantization of a fine-tuned UltraBERT-150M encoder collapsed macro-F1 from 77 to 29, and the author traced the failure to the FFN's down projection, which reproduced the collapse when quantized alone while quantizing the gate and up projections changed no predictions on a 64-article diagnostic subset. Keeping that single projection in FP32 restored most of the quality, and a subsequent serving check showed the recovered score did not mean identical behavior once request batching changed. The measurements cover Torch 2.8's x86 eager dynamic INT8 path, with embeddings and normalization left in FP32.", "body_md": "# \n[One Linear layer broke my INT8 encoder](https://julesbelveze.github.io/one-linear-layer-broke-my-int8-encoder/)\n\nMy [last two posts](https://julesbelveze.github.io/zero-shot-plateau/) argued that if your job is classification, a\nfine-tuned encoder is still hard to beat. Once I had a model that worked, the\nnext question came naturally: can I make it smaller and faster on CPU without\nlosing much quality?\n\nI went for dynamic INT8 quantization first, mostly because it's the lazy option\n(in a good way). You keep the trained checkpoint, lower the precision of the\nLinear operations, and get cheaper inference without training anything new. I\nexpected a boring tradeoff measurement. Instead, my fine-tuned UltraBERT dropped\nfrom **77 to 29 macro-F1**. It still returned labels; it just put almost all of\nthem in two classes.\n\nThis post is the trail I followed from that collapse down to a single Linear projection. Along the way, some measurements that looked useful sent me in the wrong direction, and keeping that one projection in FP32 brought back most of the quality. Then a serving check showed that a recovered score didn't mean the model behaved the same when I changed how requests were batched.\n\nThe question I kept coming back to: **what evidence would actually convince me\nthis compressed model is worth deploying?** Getting a smaller file is easy. I\nwanted to know where its predictions broke, whether a targeted fix could help,\nand what was still different once the score looked fine again.\n\n## Finding which model component breaks under INT8\n\nSame task as before: newsletter-section classification, where the model predicts\na section from an article summary. Same disclosure as before too: my dataset, my\nlabel mapping. This time I used a fresh, text-deduplicated split, so don't\ncompare these scores with the earlier leaderboard.[1](#fn:methods)\n\nI fine-tuned [UltraBERT-150M](https://huggingface.co/dmis-lab/UltraBERT-150M) and converted its Linear modules with\nPyTorch's eager dynamic INT8 path. Embeddings and normalization stayed FP32. The\nimplementation matters here: everything below is about Torch 2.8's x86 path,\nnot every way of running an encoder in INT8.\n\nMy first suspicion was boring: maybe I broke something when saving and reloading. A quantized artifact has to rebuild the right module types and packed weights, so a reload bug would easily explain a big drop. It wasn't that. The reloaded model reproduced its pre-export predictions. Changing the batch size or removing padding didn't bring the quality back either.\n\nConverting every Linear module at once changed too many things to give me a lead. So I picked a fixed diagnostic subset of 64 validation articles and quantized groups of modules one at a time, always starting from the same FP32 checkpoint. The goal was to find one group that reproduced the collapse on its own before going layer by layer.\n\nQuantizing only attention stayed close to FP32. Quantizing the feed-forward networks (FFNs) caused a big drop. Keeping the classifier in FP32 didn't rescue anything. So the FFNs alone were enough to trigger the problem, without touching attention or the classifier. That's where I went digging.\n\n## Testing weight rounding inside the FFN\n\nUltraBERT's FFN has two branches. The gate projection goes through a SiLU and gets multiplied element by element with the up projection. Then a down projection maps the product back to the hidden width.\n\n```\nhidden = silu(gate_proj(x)) * up_proj(x)\noutput = down_proj(hidden)\n```\n\nAll three are Linear layers, but they don't see the same thing. Gate and up get the incoming hidden state; down gets the product of the two branches. That made me want to test them separately instead of treating the whole FFN as one block.\n\nQuantizing the gate and up projections didn't change a single prediction on the diagnostic subset. Quantizing the down projections reproduced the failure. That told me where to look, not why. It doesn't show that the multiplication, or anything specific about its output, is the cause.\n\nNext I tried the simplest explanation I could think of: maybe rounding the FFN weights is just that damaging. I rounded them to their INT8 values, dequantized them, and ran everything in FP32. Same weight approximation, no dynamic INT8 execution. Quality stayed close to the reference.\n\nThat distinction turned out to matter. [Dynamic quantization](https://pytorch.org/blog/quantization-in-practice/) doesn't\nonly pre-quantize weights; it also quantizes activations at runtime. An\noperation can take and return float tensors and still do different arithmetic\ninternally than an FP32 operation with rounded weights. If you only look at the\nstored weights, you miss that part.\n\nSo weight rounding alone wasn't enough to cause the collapse. Runtime quantization and kernel execution played a big part, but this experiment can't tell activation quantization apart from backend arithmetic effects. It narrowed the question; it didn't answer it.\n\n## Comparing tensor error with prediction changes\n\nThe other obvious approach was to measure how much error each quantized FFN operator introduces and start with the worst one. To keep that fair, I fed each operator the exact input it gets in the FP32 model, so its error wouldn't already include errors coming from quantized operators upstream.\n\nThe largest local error was in an **up projection** much later in the network.\nYet quantizing every up projection hadn't changed a single prediction.\n\nThat one stuck with me. A perturbation in the middle of the network still has to travel through everything after it, and its size tells you nothing about whether it will reorder the final class scores. A classifier can absorb a big numerical change and still pick the same label, while a smaller one can push a decision across a class boundary. Local error and end-to-end interventions answer different questions, and I'd been treating them as the same one.\n\nSo I went back to interventions and quantized whole FFN blocks one at a time. The first block alone caused a big drop; each of the others barely moved the score. Now I had two clues from two independent sets of experiments: the down projections were sensitive, and the first FFN block was sensitive.\n\nThe spot where they overlap was the obvious suspect, but I still had to test it directly:\n\n```\nultrabert.layers.0.mlp.down_proj\n```\n\n## Recovering quality with one FP32 projection\n\nI ran two experiments on that projection, each answering a different question:\n\n- **Quantize only that projection.** Is converting it enough, on its own, to\nbreak the model?\n- **Quantize everything except that projection.** Can the rest of the INT8\nmodel work fine with that one exception?\n\nBoth came back the way I hoped. Quantizing just the first down projection caused a big drop. Keeping it in FP32 while quantizing the other 168 Linear modules recovered most of the lost quality.\n\n*All four variants use the same 64 diagnostic articles and start from the same\nFP32 checkpoint.*\n\nThis was much more useful than knowing which tensor had the biggest error: one localized change explained both the failure and the recovery. The recovery wasn't complete, and it doesn't prove this is the only sensitive operation, but it gave me a concrete recipe to test on more data.\n\n## Checking whether batch size changes the predictions\n\nBy this point the 64-article subset had guided several of my choices, so it couldn't also be what confirmed them. I froze the exemption and evaluated it on the rest of validation, which I hadn't used for the diagnosis. (The original checkpoint selection used the full validation set, so this still isn't a clean holdout.)\n\nMacro-F1 stayed within about a point of FP32 across every batching and padding policy I tried. Good news, but one question kept bugging me: was the model making the same decisions on each article, or just getting roughly the same number right?\n\nFor serving, that difference matters a lot. Batch size is usually an infrastructure knob you turn for throughput. If the classifier feeds a routing system and its label changes when you turn that knob, the same request can go down a different path. So instead of assuming a close aggregate score meant close behavior, I compared the decisions directly.\n\nAt fixed padding, going from batches of 16 to one article at a time changed\n**67 of 1,156 INT8 decisions**. FP32 changed none. The funny part is that the\nscore actually went up a bit: processing articles alone fixed 33 mistakes\nagainst the dataset labels but introduced 28 new ones. The remaining six\nswitched between two labels that were both wrong.\n\nSo a slightly higher score was hiding a mix of fixes and regressions, and the serving policy decided which requests got which answer. That's why the score alone couldn't settle whether to deploy.\n\nHere's the first changed example in validation order, a summary about GitHub Copilot. In a batch it's Headlines & News; alone it becomes Guides & Resources. Same summary, same checkpoint, same padding length.\n\nTry another article, switch the processing policy, or compare with FP32. The\nbars show normalized model scores, not calibrated confidence.\n[Open the full explorer →](https://julesbelveze.github.io/experiments/ultrabert-batching/)\n\nTo be clear, this doesn't blame any specific article it was batched with: moving from a batch to a single forward pass also changes the shape of the computation. What it does show is that this recipe's decisions depend on the batching policy. FP32 on the same inputs is the control, and the INT8 comparison shows behavior the aggregate score was hiding.\n\nAnd the reload check I started with couldn't have caught this. It verifies that the saved model reproduces its own behavior under one policy, which is useful, but it says nothing about how the model behaves under another.\n\n## Measuring the size and latency tradeoff\n\nAfter all that, I still owed myself an answer to the original question: what did the exemption actually buy in size, latency and quality?\n\nOn the final paired CPU comparison, the deployment artifact was **about 55%\nsmaller** than FP32, with **23% lower median latency** at actual input length,\nfor less than a point of macro-F1 under that policy. That makes it a reasonable\ncandidate on these numbers, though its sensitivity to batching still has to be\nweighed against whatever it ends up being used for.[2](#fn:measurement)\n\nNote that the size and speed gains are very different. Size is what you store; latency is the work of running the model. I didn't isolate per-kernel costs, so I can't say exactly where the time went, but halving the file clearly doesn't halve the forward pass.\n\nHonestly, padding was the bigger speed lever in this run: removing fixed-length padding more than halved FP32 median latency. Had I compared padded FP32 with unpadded INT8, I'd have credited quantization for a gain that mostly came from processing less padding. Keeping the serving policy matched made the (smaller) quantization gain easier to read.\n\nBefore adopting the exemption, I'd also benchmark a smaller FP32 model. The small Ettin baseline was much faster in the original comparison, at a quality cost. Whether that's acceptable depends on the task, but compression should earn the extra moving parts it brings.\n\n## What to check before deploying a quantized model\n\nThis started as a compression experiment, and what I mostly got out of it is a way to judge the model that comes out the other end. Targeted interventions showed where the conversion damaged the checkpoint and whether one exception could fix it. The serving comparison showed why that fix had to be checked decision by decision, not just with an overall score.\n\nI put the conversion, exemption and reload checks in a [standalone bert-squeeze\nexample](https://github.com/JulesBelveze/bert-squeeze/blob/539aeaed5fa96789c3c1e7c3c6dcd505e9adba79/examples/selective_quantization/README.md) that works with a public demo checkpoint or your own\nclassifier. The UltraBERT checkpoint I measured isn't public yet, so you can't\nreproduce these exact scores for now. And the projection I kept in FP32 is a\nfinding about this checkpoint, not a default I'd copy to another encoder.\n\nI still don't know the precise numerical reason that layer is so sensitive. What I'll take into the next deployment is the process: isolate one change, check that the fix really recovers the behavior you care about, then measure under the policy you actually plan to serve. A smaller model with a recovered benchmark score is a candidate, not a verdict, and the individual predictions are part of validating it.\n\n1. \nThe [methods and recorded results](https://julesbelveze.github.io/experiments/ultrabert-batching/evidence/) cover the split, label mapping, ablations and runtime pins. The newsletter\ntaxonomy has ambiguous mappings, and some source articles still overlap\nafter text deduplication. This is a numerical case study on one checkpoint,\nnot evidence that it generalizes to future newsletters. I didn't evaluate\nthe rescued recipe on test data.[↩](#fnref:methods)\n2. \nThe paired CPU measurements used the same worker, inputs and thread settings. They don't establish production tail latency or runtime memory use. A fresh-runtime check matched the final validation scores but differed on two diagnostic INT8 scores and on timings. I kept both observations on the methods page; I haven't figured out the cause yet. [↩](#fnref:measurement)", "url": "https://wpnews.pro/news/one-linear-layer-broke-my-int8-encoder", "canonical_source": "https://julesbelveze.github.io/one-linear-layer-broke-my-int8-encoder/", "published_at": "2026-10-30 00:00:00+00:00", "updated_at": "2026-11-16 13:23:26.013095+00:00", "lang": "en", "topics": ["machine-learning", "natural-language-processing", "ai-infrastructure", "mlops"], "entities": ["UltraBERT-150M", "PyTorch", "Torch 2.8", "UltraBERT"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/one-linear-layer-broke-my-int8-encoder", "markdown": "https://wpnews.pro/news/one-linear-layer-broke-my-int8-encoder.md", "text": "https://wpnews.pro/news/one-linear-layer-broke-my-int8-encoder.txt", "jsonld": "https://wpnews.pro/news/one-linear-layer-broke-my-int8-encoder.jsonld"}}