Here’s the sentence that should stop you mid-scroll: a compressed, 4-bit model that outperforms its full-precision parent. Not “close to.” Not “acceptable loss.” Better.
We’ve written before about the VRAM tax you pay to run anything decent locally – 140GB for a 70B model in FP16, which is why quantization exists in the first place. You shrink the weights, you pay in accuracy, you hope the tradeoff is worth it. That’s been the deal since day one. A new paper says the deal just changed.
The recipe
A team including researchers from Multiverse Computing published Quantization-Aware Healing, or QAH – a recovery method for models that have been both structurally compressed (fewer parameters) and quantized down to 4 bits. That’s two compounding cuts, and the paper’s framing is blunt about what they do to a model: they degrade reasoning, math, coding, and long-context behavior enough that you need a healing stage before you’d ship the thing.
The industry-standard fix for that has been quantization-aware training, QAT – bake the quantization noise into training itself so the model learns to compensate. It works, but it’s slow and finicky. The QAH team skipped it. Instead, according to the paper’s abstract, they distill directly from the original uncompressed model into the compressed one. The teacher is the full-precision model. The student is the 4-bit version. The student’s job isn’t to relearn the task from scratch – it’s to match the teacher’s behavior while operating inside a much smaller weight space.
That distinction matters more than it sounds like. QAT tries to make quantization hurt less during training. QAH treats quantization as done and then heals what broke, using the original model as the ground truth the whole way through. The paper reports this converges faster and more stably than QAT – a straightforwardly practical win before you even get to the headline result.
Why “outperforms” isn’t a typo
The genuinely strange part is the outcome, not just the efficiency of getting there: the healed 4-bit model doesn’t just recover lost ground, it comes out ahead of the original. If you’ve spent time picking between Q4_K_M, Q5_K_M, and Q8_0 quantization formats for local inference, you know the mental model everyone operates on: higher precision is a ceiling, quantization always gives some of it back. QAH says that ceiling was never fixed – it was a function of how the compressed model was trained, not an inherent property of the bit width.
That reframes what “4-bit” means. It’s not automatically “smaller, worse, cheaper.” Distillation-based healing turns the compression step into an opportunity to re-teach the model, and if the teacher signal is good enough, the student can end up sharper than the model it was copied from.
What this means if you’re running local models
If you’re serving models cheaply – which is the exact framing the QAH paper opens with – this is the direction to watch. A model that’s structurally smaller and 4-bit and still beats FP16 on the tasks you care about isn’t a compromise. It’s just a better model that happens to fit in less memory. We’ve talked about squeezing more context out of a fixed VRAM budget with KV cache quantization; QAH is the same instinct applied to the weights themselves – stop treating compression as a tax and start treating it as a training signal you can optimize against. If distillation-based healing generalizes past this paper, “full precision” stops being the ceiling anyone quotes.