{"slug": "int8-quantization-for-an-llm-engine-measured-honestly", "title": "INT8 Quantization for an LLM Engine, Measured Honestly", "summary": "A developer quantized Qwen3-0.6B's weights to INT8 using per-row scalar quantization, reducing weight matrices from 1761.61 MB to 441.78 MB (3.99x smaller) with worst-tensor relative error of 1.2017%, but found the true whole-model memory saving is less because the 622 MB embedding table remains f32 and the loader duplicated lm_head.weight. The technique pulls the scale factor out of the inner dot-product loop, cutting multiplications from 3.1 million to 3072 per output element, and uses round-to-nearest-even to avoid systematic downward bias across 28 layers.", "body_md": "I did per-row scalar quantization once already, on Lattice — my vector database — compressing stored vectors down to int8 so the index takes a quarter of the memory. This time it’s the same technique, aimed at something with a very different failure mode: the actual weights of an LLM, not the vectors it searches over.\n\nI want to walk through what carried over, what didn’t, and the number I almost led with before double-checking it turned out to be the wrong one.\n\nRight now my engine loads Qwen3–0.6B’s weights as f32. The file itself ships in bf16, so every weight gets widened on load — simpler math, but it means the “0.6B parameter model” actually sits at somewhere north of 2.4GB in memory before you’ve generated a single token. INT8 cuts each weight to a quarter the size. The question worth answering honestly wasn’t “does this save memory” — obviously it does — it was “how much does it cost, and does the model still work.”\n\nEach row of a weight matrix gets its own scale factor, rather than one scale for the whole matrix. The reason is that output channels genuinely differ in magnitude — some rows in a projection matrix carry values an order of magnitude smaller than others. A single scale sized to fit the loudest row crushes every quiet row down into a handful of usable int8 steps.\n\nI checked this wasn’t just theoretical. Built a synthetic weight matrix with half its rows scaled 50x smaller than the other half — the exact situation per-row scaling exists to handle — and measured both approaches:\n\n```\nper-row scales : rel RMS error = 0.80%one global scale: rel RMS error = 1.33%\n```\n\nReal gap, not a rounding difference. The cost is four bytes of extra storage per row, which on a [3072, 1024] matrix is 12KB against 3MB of weights — free, essentially.\n\nNo zero-point offset — zero maps to exactly zero, which fits transformer weights well since they’re roughly centered around it already. Asymmetric quantization handles skewed distributions slightly better but adds a term to every dot product, and the skew isn’t there to justify it.\n\nThe detail I hadn’t thought about until I wrote it: (int)value truncates toward zero, which biases *every single weight* slightly downward. Across 28 stacked layers, a consistent directional bias compounds. Round-to-nearest-even has no such bias - lrintf instead of a cast, and it's the difference between random noise and a systematic drift that gets worse the deeper the stack goes.\n\nThis is the part that makes quantized inference actually make sense rather than just being a storage trick:\n\n```\nsum_k a[k] * w[j][k] == scale[j] * sum_k a[k] * q[j][k]\n```\n\nThe scale can be pulled out of the inner loop and applied once per output element instead of once per multiply-accumulate. On a [1, 1024] × [3072, 1024] matmul that’s 3072 multiplications instead of 3.1 million.\n\nSynthetic tests are a start, not the answer. I ran the actual quantizer over Qwen3–0.6B’s real weights — all 196 projection matrices across 28 layers:\n\n```\nacross all quantized weight matrices:  f32:  1761.61 MB  int8: 441.78 MB  3.99x smaller  worst tensor: model.layers.19.mlp.down_proj.weight at 1.2017%\n```\n\nEvery one of the 196 tensors landed under 1.3% relative error, tightly matching the synthetic tests — real weights quantize just as cleanly as the Gaussian approximation predicted, which is itself a small useful result: I didn’t need to special-case anything for the real distribution.\n\n3.99x is a good headline. It’s also not the true whole-model number, and I caught that before writing it anywhere permanent.\n\nThe embedding table turns out to be the single largest tensor in the entire model — 622MB on its own — and it stays f32 on purpose, since embeddings are more precision-sensitive and comparatively small individually. Once you count it, “whole model” and “quantized matrices” stop meaning the same thing.\n\nThen I found something worse while doing that math: my loader was loading lm_head.weight as a full separate copy of embed_tokens.weight, even though the config says tied_embeddings: true - meaning the two are numerically identical. An unrelated bug from days earlier, quietly costing an extra 622MB for nothing, that I only noticed because I was being careful about a completely different number.\n\n``` php\nAs currently loaded (with the duplicate):  3006.5 MB -> 1686.7 MB   (1.78x)If the lm_head duplication were also fixed: 2384.2 MB -> 1064.3 MB  (2.24x)\n```\n\nThe honest number is 1.78x today, 2.24x once that bug’s fixed — not 3.99x. I’d rather publish the number that’s actually true than the one that photographs better.\n\nSame prompt, same greedy sampling, f32 weights against int8 weights:\n\nIdentical. Word for word. That’s the real bar quantization has to clear — not “does the error percentage look small,” but “does the model still say the same thing.”\n\nDecode throughput barely moved — 1.83 to 1.88 tokens per second, essentially flat. Worth saying plainly rather than letting the memory number imply a speed win that isn’t there: the current quantized matmul converts each int8 weight back to float before multiplying, with no vectorized int8 dot product underneath it. Today’s result is memory, full stop. A genuinely faster quantized matmul is separate work, not something this claims.\n\nThe technique is identical — per-row int8, symmetric, same math. What’s different is what’s actually at risk when it goes wrong. Lattice quantizes the *data being searched*; if the error creeps up, recall quietly drops — you get slightly worse search results. Here it’s quantizing the *weights doing the computing*; error doesn’t just sit there, it compounds through 28 layers of matrix multiplication before it ever reaches an output. Same tool, genuinely different failure mode, which is why the identical-output test mattered more here than a recall benchmark ever needed to on Lattice.\n\nModel runs correctly at a quarter the weight memory for the parts that matter most, on real hardware, with a number I trust because I checked it against a bug it accidentally exposed. Next up is CUDA — the kernels are already written and verified on a T4, the actual wiring into the forward pass is what’s left, and that’s where the quantized matmul finally gets to be fast instead of just small.\n\n*Originally published at **https://amankarki.hashnode.dev** on September 3, 2026.*\n\n[INT8 Quantization for an LLM Engine, Measured Honestly](https://blog.stackademic.com/int8-quantization-for-an-llm-engine-measured-honestly-bd859464c9c9) was originally published in [Stackademic](https://blog.stackademic.com) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/int8-quantization-for-an-llm-engine-measured-honestly", "canonical_source": "https://blog.stackademic.com/int8-quantization-for-an-llm-engine-measured-honestly-bd859464c9c9?source=rss----d1baaa8417a4---4", "published_at": "2026-09-03 11:54:40+00:00", "updated_at": "2026-09-03 12:23:27.295183+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-infrastructure"], "entities": ["Qwen3-0.6B", "Lattice"], "alternates": {"html": "https://wpnews.pro/news/int8-quantization-for-an-llm-engine-measured-honestly", "markdown": "https://wpnews.pro/news/int8-quantization-for-an-llm-engine-measured-honestly.md", "text": "https://wpnews.pro/news/int8-quantization-for-an-llm-engine-measured-honestly.txt", "jsonld": "https://wpnews.pro/news/int8-quantization-for-an-llm-engine-measured-honestly.jsonld"}}