I did per-row scalar quantization once already, on Lattice — my vector database — compressing stored vectors down to int8 so the index takes a quarter of the memory. This time it’s the same technique, aimed at something with a very different failure mode: the actual weights of an LLM, not the vectors it searches over.
I want to walk through what carried over, what didn’t, and the number I almost led with before double-checking it turned out to be the wrong one.
Right now my engine loads Qwen3–0.6B’s weights as f32. The file itself ships in bf16, so every weight gets widened on load — simpler math, but it means the “0.6B parameter model” actually sits at somewhere north of 2.4GB in memory before you’ve generated a single token. INT8 cuts each weight to a quarter the size. The question worth answering honestly wasn’t “does this save memory” — obviously it does — it was “how much does it cost, and does the model still work.”
Each row of a weight matrix gets its own scale factor, rather than one scale for the whole matrix. The reason is that output channels genuinely differ in magnitude — some rows in a projection matrix carry values an order of magnitude smaller than others. A single scale sized to fit the loudest row crushes every quiet row down into a handful of usable int8 steps.
I checked this wasn’t just theoretical. Built a synthetic weight matrix with half its rows scaled 50x smaller than the other half — the exact situation per-row scaling exists to handle — and measured both approaches:
per-row scales : rel RMS error = 0.80%one global scale: rel RMS error = 1.33%
Real gap, not a rounding difference. The cost is four bytes of extra storage per row, which on a [3072, 1024] matrix is 12KB against 3MB of weights — free, essentially.
No zero-point offset — zero maps to exactly zero, which fits transformer weights well since they’re roughly centered around it already. Asymmetric quantization handles skewed distributions slightly better but adds a term to every dot product, and the skew isn’t there to justify it.
The detail I hadn’t thought about until I wrote it: (int)value truncates toward zero, which biases every single weight slightly downward. Across 28 stacked layers, a consistent directional bias compounds. Round-to-nearest-even has no such bias - lrintf instead of a cast, and it's the difference between random noise and a systematic drift that gets worse the deeper the stack goes.
This is the part that makes quantized inference actually make sense rather than just being a storage trick:
sum_k a[k] * w[j][k] == scale[j] * sum_k a[k] * q[j][k]
The scale can be pulled out of the inner loop and applied once per output element instead of once per multiply-accumulate. On a [1, 1024] × [3072, 1024] matmul that’s 3072 multiplications instead of 3.1 million.
Synthetic tests are a start, not the answer. I ran the actual quantizer over Qwen3–0.6B’s real weights — all 196 projection matrices across 28 layers:
across all quantized weight matrices: f32: 1761.61 MB int8: 441.78 MB 3.99x smaller worst tensor: model.layers.19.mlp.down_proj.weight at 1.2017%
Every one of the 196 tensors landed under 1.3% relative error, tightly matching the synthetic tests — real weights quantize just as cleanly as the Gaussian approximation predicted, which is itself a small useful result: I didn’t need to special-case anything for the real distribution.
3.99x is a good headline. It’s also not the true whole-model number, and I caught that before writing it anywhere permanent.
The embedding table turns out to be the single largest tensor in the entire model — 622MB on its own — and it stays f32 on purpose, since embeddings are more precision-sensitive and comparatively small individually. Once you count it, “whole model” and “quantized matrices” stop meaning the same thing.
Then I found something worse while doing that math: my was lm_head.weight as a full separate copy of embed_tokens.weight, even though the config says tied_embeddings: true - meaning the two are numerically identical. An unrelated bug from days earlier, quietly costing an extra 622MB for nothing, that I only noticed because I was being careful about a completely different number.
As currently loaded (with the duplicate): 3006.5 MB -> 1686.7 MB (1.78x)If the lm_head duplication were also fixed: 2384.2 MB -> 1064.3 MB (2.24x)
The honest number is 1.78x today, 2.24x once that bug’s fixed — not 3.99x. I’d rather publish the number that’s actually true than the one that photographs better.
Same prompt, same greedy sampling, f32 weights against int8 weights:
Identical. Word for word. That’s the real bar quantization has to clear — not “does the error percentage look small,” but “does the model still say the same thing.”
Decode throughput barely moved — 1.83 to 1.88 tokens per second, essentially flat. Worth saying plainly rather than letting the memory number imply a speed win that isn’t there: the current quantized matmul converts each int8 weight back to float before multiplying, with no vectorized int8 dot product underneath it. Today’s result is memory, full stop. A genuinely faster quantized matmul is separate work, not something this claims.
The technique is identical — per-row int8, symmetric, same math. What’s different is what’s actually at risk when it goes wrong. Lattice quantizes the data being searched; if the error creeps up, recall quietly drops — you get slightly worse search results. Here it’s quantizing the weights doing the computing; error doesn’t just sit there, it compounds through 28 layers of matrix multiplication before it ever reaches an output. Same tool, genuinely different failure mode, which is why the identical-output test mattered more here than a recall benchmark ever needed to on Lattice.
Model runs correctly at a quarter the weight memory for the parts that matter most, on real hardware, with a number I trust because I checked it against a bug it accidentally exposed. Next up is CUDA — the kernels are already written and verified on a T4, the actual wiring into the forward pass is what’s left, and that’s where the quantized matmul finally gets to be fast instead of just small.
Originally published at https://amankarki.hashnode.dev on September 3, 2026.
INT8 Quantization for an LLM Engine, Measured Honestly was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.