A quantization compiler that delivers 12× lower KL than Q4_K_M at the same size A quantization compiler called LookAtMyQuant produces GGUF files with 12 times lower KL divergence from the original model than llama.cpp's Q4_K_M at the same file size, according to the project's own measurements on Gemma 4 E2B. At Q4_K_M's size, the compiled file solves 59% of GSM8K problems versus 50% for Q4_K_M and 61% for the unquantized original, with most of the gain coming from storing Gemma's 2.35-billion-weight per-layer lookup table at 4.25 bits instead of 6.56 and reallocating the saved bytes to weight matrices. On Qwen3.5-4B and SmolLM2-360M, the compiler's plans showed 6 to 81 times lower KL than 3-bit models that keep the embedding or KV cache at 16 bits at equal cost on each target device. A quantization compiler that delivers 12× lower KL than Q4 K M at the same file size Abstract Most quantized language models use one format and one bit width for every tensor, with a few hand-written exceptions. LookAtMyQuant is a quantization compiler. It measures how much each stored piece of a model hurts the whole model when that piece alone is compressed. The pieces are every weight matrix, buckets of embedding rows grouped by token frequency, and each layer's key–value cache. It measures a few codes for each piece and predicts the rest. Then it gives each piece the code that keeps total damage lowest while the model fits the size or speed budget of a target device. Each code is a short program in a grammar of tensor-network rewrites. Nine published quantization and pruning methods are short programs in this grammar, reproduced exactly. On Gemma 4 E2B, the compiler restricted to llama.cpp's types makes ordinary GGUF files. At the size of llama.cpp's Q4 K M, the compiled file has 12 times lower KL divergence from the original model, and it solves 59% of GSM8K problems against 50% for Q4 K M and 61% for the original. Most of the gain comes from Gemma's 2.35-billion-weight per-layer lookup table. It is cheap to compress, so the compiler stores it at 4.25 bits instead of 6.56 and gives the saved bytes to the matrices. On Qwen3.5-4B and SmolLM2-360M, compared with 3-bit models that keep the embedding or the cache at 16 bits, the compiler's plans have 6 to 81 times lower KL at equal cost on each target device. A quantization format applies one rule to a whole model. GGUF's Q4 K M, for example, stores most matrices at about 4.5 bits per weight and a fixed list of tensors at Q6 K. The rule is the same for a 360-million-parameter model and a 70-billion-parameter one, for a model with a large vocabulary and one with a small one, and for a phone and a server. But the damage each piece does to the model is not uniform. In Gemma 4 E2B, one 0.4-million-weight matrix costs nearly as much at 3 bits as twenty whole layers. Half of the model's weights are a lookup table that tolerates 4 bits well. The 256 most frequent tokens make up a third of all text, while the rarest half of the vocabulary makes up a fraction of a percent. A fixed recipe ignores these differences. LookAtMyQuant makes a plan that names one code for every stored piece of the model. It finds the plan that does the least damage within a given budget on a given device. We make four contributions: We compared the compiler's files with every widely used quantization of Gemma 4 E2B that we could load. These are Google's quantization-aware-trained q4 0 GGUF, unsloth's UD-Q4 K XL, bartowski's Q4 K M and LM Studio's MLX 4-bit. We also built llama.cpp's stock recipes, Q2 K to Q8 0, with an importance matrix from the same calibration text the compiler uses. Every file is turned back into full-precision weights inside the same Hugging Face model, so the same code scores every file on the same text. The text is 64 windows of WikiText-2 and 8 held-out windows from each of 13 sources: English, ten other languages, code and chat. Each window has 512 tokens and starts with the beginning-of-sequence token. KL divergence measures how far a file's next-token predictions are from the original model's; lower is closer. Sizes count the bytes of the text model's weights, for every file. Table 1. Each compiled GGUF against the public files within 0.06 GB of its size. Google's QAT file is left out here because Google trained it further after quantizing, so its KL includes that training; Table 2 compares it on benchmarks. | Compiled GGUF | KL | Public file | Size GB | KL | KL ratio | |---|---|---|---|---|---| The gap holds on every one of the 13 sources and on WikiText-2, which is not part of the calibration text. On WikiText-2 the 3.39 GB compiled file has a KL of 0.008, and Q4 K M has 0.143. Multiple-choice benchmarks show small differences. Except for Q2 K and Google's QAT file, every file scores within about two points of the original on MMLU, ARC-Challenge and HellaSwag. On GSM8K, a multi-step math benchmark, public 4-bit files lose up to 27 points, and compiled files at most 6 Figure 2 . With 250 problems, each GSM8K figure is uncertain by about ±6 points. Gemma 4 E2B feeds each of its 35 layers a 256-wide vector looked up by token. Together these vectors form a table of 262,144 rows by 8,960 columns, which holds 2.35 billion of the model's 4.63 billion text weights. The table is never multiplied, only read one row per token. The compiler measured it as cheap to compress. Per weight, it costs less than a tenth as much as the least sensitive matrices. llama.cpp's Q4 K M recipe stores this table at Q6 K, 6.56 bits per weight. That is 1.93 GB, 56% of the file. The compiler, choosing among the same llama.cpp types, stores it at IQ4 XS, 4.25 bits, and gives the 0.68 GB it saves to the matrices, most of which it stores at Q8 0 or BF16 Figure 3 . Figure 4 replays the compiled file one piece at a time. The compiler's native menu also has trellis codes and Lloyd–Max codes. Both run on Apple Silicon with our MLX runtime https://github.com/ShinMegamiBoson/lookatmyquant/tree/main/runtime mlx . With both kinds of code, its 2.00 GB plan has a KL of 0.074, lower than every public file under 3.5 GB, and solves 60.4% of GSM8K problems. At 3.41 GB its KL is 0.0057. Table 2. Every Gemma 4 E2B file we measured. KL on the 13-source text mix and on WikiText-2; accuracy on MMLU 1,000 sampled questions, zero-shot , ARC-Challenge all 1,172 , HellaSwag 1,000 sampled and GSM8K the first 250, five-shot . | File | Size GB | Bits / weight | KL, text mix | KL, WikiText | MMLU | ARC-C | HellaSwag | GSM8K | |---|---|---|---|---|---|---|---|---| Qwen3.5-4B uses its 248,320-row embedding table both to look up input tokens and as the output head. At 16 bits the table takes about as many bytes as all the matrices at 3 bits. At the same size as a model with 3-bit matrices and a 16-bit embedding, the compiler's plan stores the embedding at 3.1 bits on average and the matrices at 5.35. Perplexity rises 0.13% instead of 4.9%, and KL is 12 times lower Table 3 . The plan gives the 256 most frequent rows 16 bits, the next 8,000 rows 8 bits, and the rarest 117,000 rows 2 bits. The 183,000 rows whose tokens never appear in WikiText's training split cost 0.0007 KL in total when all of them are stored at 2 bits. Table 3. Qwen3.5-4B, 64 WikiText-2 test windows of 512 tokens. Original perplexity 12.283. Each compiler plan costs the same as the uniform plan named in the second column. | Plan | Same cost as | Matrix bits | Embedding bits | Cache bits | Perplexity | KL | |---|---|---|---|---|---|---| | 1. Uniform 3-bit trellis, 16-bit embedding | – | 3.06 | 16 | – | 12.884 | 0.0614 | | 2. Compiler, widths for the matrices only | Plan 1, in file size | 3.05 | 16 | – | 12.848 | 0.0592 | | 3. Compiler, matrices and embedding | Plan 1, in file size | 5.35 | 3.10 | – | 12.299 | 0.0050 | | 4. Uniform 3-bit trellis, 3-bit embedding | – | 3.06 | 3.04 | – | 13.008 | 0.0735 | | 5. Compiler, matrices and embedding | Plan 4, in file size | 3.17 | 2.41 | – | 12.801 | 0.0542 | | 6. Compiler for a laptop with a 32,768-token context | Plan 1 with a 16-bit cache, in bits read per token | 6.62 | 4.23 | 6.19 | 12.298 | 0.0015 | What a code costs depends on the device. On disk, only stored bits count. On a laptop generating one sequence, the weights and the key–value cache are both read for every token, so cache bits count too. On a server running 32 sequences, one read of the weights serves all of them, so weights become cheap and the cache dominates. On a device limited by compute, decode work counts. For SmolLM2-360M the compiler makes a different plan for each Table 4 . The server plan stores matrices at 7.4 bits because 32 sequences share each read, and the compute-limited plan avoids trellis codes, which take more work to decode. Table 4. SmolLM2-360M. Each compiler plan costs the same as its reference plan on that device. Cells are WikiText-2 perplexity / KL; the laptop, server and compute-limited rows use windows of 2,048 tokens. | Device | Reference plan | Reference | Compiler | What the compiler changed | |---|---|---|---|---| | Disk | 3-bit trellis matrices, 16-bit embedding | 16.72 / 0.0959 | 15.48 / 0.0128 | Embedding from 16 to 4.8 bits; matrices from 3.0 to 4.7 bits | | Disk | 3-bit trellis matrices and embedding | 17.40 / 0.1376 | 16.81 / 0.1068 | Embedding rows by frequency, 8 bits for the most frequent to 2 bits for the rarest | | Laptop: one sequence, 2,048-token context | 3-bit trellis, 16-bit embedding and cache | 12.25 / 0.0979 | 11.22 / 0.0050 | Cache keys 8 bits, values 5–6; matrices 5.8 bits; embedding 5.6 bits | | Server: 32 sequences, 16,384-token context | 3-bit trellis, 16-bit embedding, 4-bit cache | 12.70 / 0.1323 | 11.44 / 0.0216 | Matrices 7.4 bits; cache keys 4–5 bits, values 3 | | Compute-limited device | 3-bit trellis, 16-bit embedding and cache | 12.25 / 0.0979 | 11.20 / 0.0012 | Lloyd–Max codes at 6–8 bits instead of trellis codes; cache at 6–8 bits | The large gains come from pieces that uniform formats leave at 16 bits or treat as one block: the embedding table, the per-layer table and the cache. On top of a strong base quantizer, choosing a width per matrix helps little. On SmolLM2-360M, at the same average bits as the uniform 3-bit and 4-bit programs, per-matrix widths lower KL by 3% and 5% Table 5 . Which layer a matrix is in matters more than what kind of matrix it is. Keeping the widths chosen for each kind of matrix but assigning them to random layers gives 21% more KL than the uniform program. Table 5. SmolLM2-360M, matrices only; embedding and cache at 16 bits. Original perplexity 15.31. Cells are perplexity KL . | Bits per weight | Uniform trellis | Compiler | Same widths, shuffled across layers | |---|---|---|---| | 2.05 | 23.46 0.439 | – | – | | 2.25 | – | 20.50 0.308 | – | | 2.50 | – | 18.63 0.213 | 19.82 0.267 | | 3.04 | 16.72 0.0959 | 16.64 0.0926 | 16.98 0.1163 | | 3.50 | – | 15.95 0.0484 | – | | 4.04 | 15.64 0.0237 | 15.61 0.0226 | – | LookAtMyQuant writes a model as a tensor network. Each weight matrix is a node with named indices, and nodes that share an index are joined. A quantization method is a sequence of rewrites of this network, built from five kinds of word, named after parts of speech. Figure 5 shows one layer's MLP in this form. Nine published methods are short programs in this grammar, each reproduced exactly, layer by layer, against its reference code: RTN, GPTQ, AWQ, SVD, SVD-LLM, SliceGPT, magnitude pruning, Wanda and SparseGPT. The same words also build MXFP4, NVFP4, QuIP -style codebooks and QTIP-style trellis codes. Figure 6 steps through the program the compiler uses most, and Figure 7 shows the rewrites behind five published methods. Table 6. The compiler's menu for a weight matrix. Bits are for a matrix 1,536 columns wide. The last column is the median, over 32 Gemma 4 E2B matrices measured with every program, of KL divided by the 3-bit trellis's KL. | Program | Lossy step | Bits per weight | KL relative to the 3-bit trellis | |---|---|---|---| The compiler runs the model once. Linear layers that read the same tensor form one stage and are quantized together. Embedding tables are cut into 11 buckets of rows ranked by how often their token appears in the calibration text: 256, 256, 512 and so on, doubling up to 131,072 rows. A table that is also the output head is read two ways, as the head at every position and as a lookup of one row, so its error is weighted by the head's inputs plus the identity. A table that is only looked up is weighted by the identity alone. Small linear layers, routers and normalization weights stay at 16 bits, and their bytes are counted. For each piece and code, the compiler quantizes that piece alone and measures the KL divergence of the model's next-token distribution from the original's, on 16 held-out windows of 512 tokens. The windows come from the calibration split, never the test split. On SmolLM2-360M we measured all 2,312 pairs of piece and code this way. For larger models, each matrix gets one cheap probe instead. The probe adds Gaussian noise shaped like the error that LDLQ leaves. LDLQ is the error-correcting rounding used by GPTQ and QuIP. The noise is scaled by the square root of LDLQ's error variances and by the inverse of the triangular factor of H, the second moment of the inputs to the matrix. This takes one forward pass and no code search. A few layers are measured with every real code. On Gemma 4 E2B, the 275 matrices got a probe each, four layers and all 22 table buckets got real codes, and measuring took about one H100-hour. The prediction for piece i and code c is KL