Quantisation and Verifying Weights

A practitioner's technical note explains how 4-bit quantisation of open-weight models works and how the resulting GGUF files can be verified, walking through a worked example in which four weights stored as 4-bit signed integers in the range −7 to 7 plus one shared scale produce a maximum error of 0.07, half the scale. The note states that 8-bit quantisation is nearly lossless, 4-bit is the common sweet spot at a bit over a quarter of the 16-bit size, and quality drops noticeably below that, and it groups methods under round-to-nearest (RTN), covering llama.cpp's Q8_0 and Q4_0, its K-quants made without an imatrix, FP8 casts, and MXFP4/NVFP4. The rounding error bound is presented as the basis for checking whether a downloaded quant is the model it claims to be.

The question that started this I run open-weight models locally: models whose files are published for anyone to download and run on their own hardware my first go is in Qwen 3.6 first impressions ../../ai/local-models/qwen-3.6-35A-3b-first-impressions . Usually I download a smaller version someone else made, such as a “4-bit GGUF”. GGUF is the file format used by llama.cpp, a popular tool for running models on ordinary computers. What “4-bit” means takes most of this note. That raises a question I couldn’t answer: how do I know the file is really the model it claims to be, and not a corrupted copy, a worse model with a famous name, or something tampered with? Answering it sent me back to basics. What a model actually is A model is mostly a giant list of numbers. Each number is a weight or parameter . When the model processes text, it multiplies and adds its way through them. Training is nudging them until the outputs are useful. The structure they live in is in Transformer Basics ../../ai/attention/transformer-basics . The weights are grouped into tensors : named grids of numbers, such as “layer 12’s attention weights”. Tensors are saved as files, and those files are what you download. A “70B” model has 70 billion weights. Computers store numbers as bits ones and zeros , commonly 16 bits per weight. Eight bits make a byte, so that’s 2 bytes each: To run at a usable speed, all of it should sit in fast memory, ideally the graphics card’s GPU’s . Even a high-end gaming GPU holds a small fraction of 140 GB, and most laptops less still. What quantisation is Quantisation means storing each number with fewer bits. A quantised model is often just called a quant . The everyday version is rounding. Round every price in a shop to the nearest 5 cents and you need fewer distinct values, with no item off by more than 2.5 cents. Or save a photo with 256 colours instead of millions: the file shrinks and the picture mostly looks the same. bits can represent different values. 16 bits gives 65,536 options per weight; 4 bits gives 16. The trick is spending those 16 wisely. The usual mechanism: a small block of weights shares one scale , stored at higher precision. Each weight becomes a small integer, and the approximate value comes back as integer × scale. That step back is dequantising . A worked example Four weights, quantised to 4-bit signed integers whole numbers that can be negative in the range −7 to 7. Four bits can hold −8 to 7; −7…7 keeps it symmetric. The scale maps the largest magnitude to the top of the range: Divide each weight by the scale and round to the nearest integer: Dequantise by multiplying back: | Original | | Stored | Dequantised | Error | |---|---|---|---|---| | 0.12 | 0.86 | 1 | 0.14 | 0.02 | | −0.53 | −3.79 | −4 | −0.56 | 0.03 | | 0.98 | 7.00 | 7 | 0.98 | 0.00 | | 0.31 | 2.21 | 2 | 0.28 | 0.03 | Four 16-bit numbers became four 4-bit integers plus one scale. The part to hold on to: every error is at most 0.07, half the scale. That isn’t luck. Rounding to the nearest integer is never off by more than half a step, and a step here is one scale: This bound comes back later as a way to check a quant. The trade-off Fewer bits means a smaller file that fits on more modest hardware. It also tends to mean faster inference running the model to generate text : producing each token, a word or piece of one, involves reading through the weights, and there are fewer bytes to read. The price is a little more error in every weight. Roughly: 8-bit is nearly lossless. 4-bit is the common sweet spot, at a bit over a quarter of the 16-bit size. Below that, quality drops noticeably. Families of methods Round-to-nearest RTN . Choose scales from the weights alone and round, as above. This covers llama.cpp’s Q8 0 and Q4 0 8-bit and 4-bit , its more refined K-quants when made without an imatrix below , FP8 casts, and MXFP4/NVFP4. FP8, MXFP4 and NVFP4 are floating-point formats: like scientific notation, the gap between representable numbers grows with their size. Calibration-based. Feed sample text calibration data through the model, see which weights matter for the outputs, and choose roundings that hurt those outputs least. GPTQ rounds one column at a time and nudges the not-yet-rounded weights to compensate for the error so far, so final values often aren’t the nearest rounding of the originals. AWQ finds the important channels groups of weights reading from inputs that tend to be large , scales them up for finer resolution, and folds the inverse scale into a neighbouring layer so the maths still works. llama.cpp’s imatrix “importance matrix” gathers similar statistics to steer how each block is quantised. Quantisation-aware training QAT . The model is trained or fine-tuned while simulating low precision, so it learns to tolerate it. Its weights have legitimately moved away from the original’s, so comparing against the original tells you little. Can I check a quant against the original? As far as I can tell there are three levels, strongest first. 1. Bit-exact reproduction A hash is a short fingerprint of a file’s bytes SHA-256 is common . Change one bit and the fingerprint changes completely, so matching hashes mean, for practical purposes, identical files. For deterministic, calibration-free methods RTN, FP8 and MXFP4 casts, Q8 0, K-quants without imatrix , I can re-quantise the originals myself and compare hashes. Same input, same algorithm, same output. A match means I don’t have to trust whoever made the download. HQQ is a near miss: no calibration data, but it optimises its scales iteratively, and results may only be near-identical across GPUs or library versions. 2. Structural check Failing that, I can dequantise each tensor and compare it element by element with the original. For RTN this is the bound from the worked example: every dequantised weight must sit within scale/2 of the original. One weight further out means the quant wasn’t made from these weights by nearest-rounding. In floating-point formats the step varies, so it’s “half a step” instead. The other families bend this: - GPTQ breaks the bound by design. - AWQ needs its scales undone first. They’re folded into neighbouring layers rather than stored, and I’m still working out how reliably they can be recovered. - imatrix quants stay near the bound but can’t be reproduced without the exact imatrix file. So for my opening 4-bit GGUF: a plain Q4 0, or a K-quant without imatrix, I can reproduce and hash. With an imatrix, I’m down to this check unless the maker published the file. 3. Behavioural only For GPTQ or AWQ without published calibration data, QAT, and methods like AQLM weights stored as entries in a learned table of patterns, a codebook and SpQR outlier weights kept at higher precision , the numbers can’t be compared directly. What’s left is comparing what the models do. At each step a model gives a probability for every possible next token. KL divergence measures how far the quant’s probabilities are from the full-precision model’s on the same text: zero means identical. llama.cpp’s perplexity tool perplexity scores how surprised a model is by text has a KL-divergence mode. A cruder check is top-token agreement: how often both pick the same most-likely token. What behavioural checks miss These catch broken quants. They don’t catch subtle tampering, like a backdoor that activates only on a specific trigger phrase. Ordinary test text never contains the trigger, so the scores look fine. Safetensors: the file format underneath All this needs original weights I can trust, and the format matters there too. Older checkpoints from PyTorch, the most widely used library for building models, are .bin or .pt files saved with Python’s pickle . Pickle saves general Python objects, and a pickle file can include instructions to run code on load. That’s the design, not a bug. Loading a pickle can run code A malicious .bin or .pt file can run arbitrary commands on your machine just by being loaded. Newer PyTorch versions load these more defensively by default, which I’m still reading up on, but the format allows it. Safetensors is deliberately dumb. The first 8 bytes give the length of a small JSON plain-text header listing each tensor’s name, data type such as 16-bit float , shape and byte offsets. Then raw bytes. No code, so nothing to execute. Side benefits: - Fast loading. The file can be memory-mapped: the operating system treats it as memory and reads pieces only when touched. Zero-copy, and you load only the tensors you need. - Easy to inspect. The header says where each tensor’s bytes sit, so you can read or hash one on its own. - The default on Hugging Face, the main site where models are published. For verification, I can hash the original safetensors against the publisher’s repository at a specific commit a fixed snapshot of its files , then read individual tensors for the structural check. Safe to load is not trustworthy Safetensors makes loading safe. It doesn’t make the weights trustworthy. A backdoored model in safetensors is still backdoored. Provenance who published it, and whether I have their exact files still matters. My practical recipe 1. Download the original safetensors from the official repository. 2. Verify their hashes against a specific commit. 3. Quantise them myself with a deterministic method: Q8 0, FP8, or K-quants without imatrix. 4. Run a KL-divergence check against the full-precision model as a sanity check. If I want imatrix, GPTQ or AWQ quality, I generate the calibration artefacts myself from the originals. This mostly sidesteps my opening question by not relying on other people’s quants. Open questions - How to detect a backdoor that survives quantisation, when behavioural checks on ordinary text never see the trigger. - Whether GPU nondeterminism kills bit-exact reproduction in practice. I expect llama.cpp’s CPU quantisation to be repeatable, but haven’t tested it across machines. - How cleanly the half-step bound holds for K-quants which search for a good scale rather than using max/7 and for FP8 and MXFP4. - Whether tool versions and file metadata break whole-file hash matches, and whether hashing tensor by tensor is the better habit. - How practical the KL check is for models too large for my memory, since it needs the full-precision model as a reference. Notation / glossary | Term | Meaning | |---|---| | | An original weight. | | | The small integer stored in its place. | | | The dequantised weight, . | | Weight | A number the model learned in training; also called a parameter. | | Bit | A single 0 or 1. bits represent different values. | | Scale | A number shared by a block; stored integer × scale gives back the approximate weight. | | Quantise | Store numbers with fewer bits by dividing by a scale and rounding. | | Dequantise | Multiply stored integers by the scale to get approximate weights back. | | Calibration | Running sample text through a model to learn which weights matter most. | | Hash | A short fingerprint of a file’s bytes; any change to the file changes it. | | Safetensors | A weights format: a JSON header plus raw bytes, with no executable code. | | Pickle | Python’s general object-saving format; loading it can run arbitrary code. |