Qwen3-4B ships as an 8.05 GB file of 16-bit weights. In this post we walk through how a 2-bit quantizer gets the same model down to 1.24 GB (85% smaller) without retraining, and how to check whether the result is any good. Our worked example is Pingala-8, an open-source format we built on llama.cpp. Most of the ideas are not ours: the rotation and rounding come from QuIP and QuIP#, the trellis from QTIP and Ungerboeck's modem codes, and ikawrakow's IQ2_KT is its closest relative. We will also show which piece mattered less than we expected.
At 2 bits per weight, each weight can take one of 4 values. Every weight is rounded to its nearest allowed value, and the difference is error. Two things make that error hurt.
It adds up. Each output of a layer is a sum of thousands of weight-times-input products. The rounding errors in those products do not cancel on their own, and they travel through dozens of layers.
Outliers. Weights are stored in small groups, each with a scale so its 4 levels cover the group's range. Take the group [3.0, 0.2, -0.1, 0.1]. To reach 3.0, the levels must be spread wide, say {-3, -1, 1, 3}. Now 0.2, -0.1 and 0.1 all land on -1 or +1, an error of about 1 each, larger than the weights themselves.
A good 2-bit quantizer must remove outliers, make errors cancel, and get more precision per bit. Those are the three tricks.
If R is an orthogonal matrix (R^T R = I), then W x = (W R^T)(R x). We can store the rotated weights W R^T, apply R on the input side, and the layer computes exactly the same result. A rotation keeps lengths but changes coordinates, so it can spread one large value across many coordinates.
The usual choice is a Hadamard matrix with random signs, because it is cheap to apply. The 4x4 version, scaled by 1/2:
[ 1 1 1 1 ]
1/2 * [ 1 -1 1 -1 ]
[ 1 1 -1 -1 ]
[ 1 -1 -1 1 ]
Apply it to [3.0, 0.2, -0.1, 0.1] and you get [1.6, 1.3, 1.6, 1.5]. The sum of squares is 9.06 both times, but the largest value is now close to the others instead of 15 to 30 times bigger, and four evenly spaced levels fit it well. Real layers rotate many coordinates at once, and the rotated weights look roughly like a bell curve without extreme outliers.
Plain rounding treats each weight on its own. The layer cares about the output, the sum of w_i * x_i.
Toy example: three weights [0.4, 0.4, 0.4] multiply inputs that are almost always equal, so the layer effectively uses their sum, 1.2. Rounding each to the nearest integer gives [0, 0, 0], off by 1.2. With feedback, the first 0.4 rounds to 0 and its error (0.4) is carried to the next weight, which becomes 0.8 and rounds to 1; that error (-0.2) is carried to the third, which becomes 0.2 and rounds to 0. The result [0, 1, 0] sums to 1, off by only 0.2.
LDLQ, from the QuIP paper, is the principled version. It quantizes the columns of a weight matrix one at a time and pushes each column's error onto the columns not yet quantized. How much goes where comes from H, the average of x x^T over inputs collected by running sample text through the model. If the inputs were uncorrelated, H would be diagonal and LDLQ would reduce to plain rounding.
IQ2_XXS, llama.cpp's smallest common 2-bit format, uses a codebook: each group of weights becomes the nearest entry of a fixed table, one group at a time. Trellis-coded quantization, used by QTIP and IQ2_KT, adds memory instead.
The encoder carries a small state from one group to the next. The state decides which subset (a "coset") of values the next group may use. The bits written for that group pick a value inside the subset and also set the next state. The decoder tracks the state for free, so the format can use twice as many values as the bits alone could address. The price: today's choice decides tomorrow's menu, so you cannot round each group on its own.
A toy with 1 bit per weight shows both sides. Four levels are split into two cosets: state 0 may use {-1.5, 0.5} and state 1 may use {-0.5, 1.5}. Bit 0 picks the lower value, bit 1 the higher, and the bit you wrote becomes the next state. Start in state 0 and encode [-0.4, 0.1, 1.5]:
| Step | Target | Greedy: state, pick | Best path: state, pick |
|---|---|---|---|
| 1 | -0.4 | 0, 0.5 | 0, -1.5 |
| 2 | 0.1 | 1, -0.5 | 0, 0.5 |
| 3 | 1.5 | 0, 0.5 | 1, 1.5 |
| Squared error | 2.17 | 1.37 |
Greedy rounding takes the closest value at step 1 and is stuck with the wrong menus at steps 2 and 3. The best path accepts a worse first step to get the right menus later. Plain 1-bit rounding to {-1, 1}, with the same 3 bits, scores 1.42: greedy trellis rounding loses to it, the searched path beats it. On real data the advantage is an average over long sequences, not a guarantee at every step.
The search is the Viterbi algorithm: dynamic programming that keeps, for every state, the cheapest path ending there. Its cost grows linearly with length, and it runs once, offline, when the file is made.
Pingala-8 uses Ungerboeck's 8-state set-partitioned trellis over symbols of 4 weights, at about 2 bits per weight. Decoding is cheap (read bits, update state, look up values), and a fused CUDA kernel does it inside the matrix-vector product. Expensive search once, cheap decode on every token: that asymmetry is why trellis coding suits inference.
Quantization results are easy to flatter by accident. Our rules, which work for any comparison:
Perplexity against file size. B4 is the released Pingala-8 file. Left: Qwen3-4B, where "post hoc" marks the tuned result. Right: Llama-3.1-8B scored without a start token in each window; the figures below add one.
Qwen3-4B, WikiText-2 perplexity at context 4096 (lower is better):
| File | Size | Perplexity |
|---|---|---|
| Original, fp16 | 8.05 GB | 12.46 |
| llama.cpp IQ2_XXS | same as ours | 28.61 |
| ik_llama.cpp IQ2_KT | same as ours | 18.15 |
| Pingala-8 | 1.24 GB | 16.64 |
On the five quizzes, Pingala-8 averages 50.8 against 41.9 for IQ2_XXS (+21%) and keeps 90% of the 8-bit Q8_0 average (56.3).
Speed for one user (batch 1), Qwen3-4B on the RTX 5070 Ti Laptop:
| Measure | vs IQ2_XXS (same size) | vs Q4_K_M (twice the size) |
|---|---|---|
| Generating text | 1.10x | 1.39x |
| Reading the prompt | 0.74x | - |
Generating one token at a time mostly means reading every weight from memory, so fewer bytes give more tokens per second while decoding stays cheap. Reading a prompt processes many tokens at once and is limited by compute, which is where the extra decode work shows.
On Llama-3.1-8B (a 2.39 GB file), Pingala-8 scores 8.44 against 9.36 for IQ2_KT at the same size.
An ablation removes one part and measures the damage. We removed the trellis memory at equal bytes. The cost was 0.076 nats per token of extra loss, about 8% higher perplexity (e^0.076 is about 1.08).
Set it against our two gaps:
The lesson for anyone building a 2-bit quantizer: rotation and error-feedback rounding carry most of the quality. Trellis memory explains most of the small margin between formats that are already close, and little of the large gap to a plain codebook format. We expected the trellis to be the main story; the ablation says otherwise, which is why it is worth running.
run_qwen3_4b.bat on Windows or ./run.sh qwen on Linux. It downloads the model once and opens a local chat page. You need an NVIDIA RTX 30, 40 or 50 series GPU.
./llama-perplexity -m model.gguf -f wiki.test.raw -c 4096 -ngl 99
./llama-bench -m model.gguf -p 512 -n 128
Pingala's Chandahsutra, an ancient Sanskrit work on poetic metre, lists the 8 patterns of three long or short syllables, the ganas. Our trellis has 8 states, so we named the format after him. The name is a credit; the trellis labels are Ungerboeck's.
Background reading: QuIP, QuIP#, QTIP, and Marcellin and Fischer's trellis-coded quantization.
This article was written with the help of AI; every number comes from our measured logs and the paper.