Shrink Your LLM by 75% and (Mostly) Keep Its Brain: Quantization Explained

A developer explains quantization techniques for large language models, including GPTQ, AWQ, GGUF, and bitsandbytes, which can reduce model size by up to 75% while preserving most accuracy. The post details how each method works and its trade-offs for GPU, CPU, or hybrid inference.

If you've ever tried to run a large language model on your own hardware, you've probably hit the same wall: the model is huge , your GPU's VRAM is not, and suddenly a 7B parameter model that "should" fit doesn't. This is where quantization comes in — and it's one of the most impactful techniques for making LLMs actually usable outside of a data center. This post breaks down what quantization actually does, the major approaches you'll run into, and how to think about the trade-offs when picking one for your project. Every parameter in a neural network is a number, and by default those numbers are usually stored as 32-bit or 16-bit floating point values FP32 or FP16/BF16 . Quantization is the process of representing those same numbers with fewer bits — commonly 8-bit, 4-bit, or even lower. Fewer bits per parameter means: The catch: reducing precision means reducing the information each number carries, which introduces error. The whole game in quantization research is minimizing that error while maximizing the compression. You might think: "just round every weight to the nearest 4-bit value and call it a day." In practice, this tanks model quality fast, for a few reasons: This is why modern quantization methods aren't just "round the numbers" — they're calibrated, often using a small dataset to figure out how to quantize with minimal damage. GPTQ Generative Pre-trained Transformer Quantization is a post-training quantization method that processes weights layer-by-layer, using second-order information an approximation of the Hessian to figure out how to round each weight while compensating for the error introduced by previous roundings. It's calibrated against a small dataset and is one of the more established 4-bit quantization methods, especially popular for GPU inference. AWQ takes a different angle: instead of trying to quantize every weight equally well, it identifies which weights are most important based on the activations that flow through them during inference, and protects those specifically often by scaling them before quantization . The insight is that a small percentage of weights matter disproportionately for output quality, and preserving those gives you most of the quality of full precision at a fraction of the size. GGUF is a file format successor to GGML commonly used with llama.cpp and tools built on it, like Ollama and LM Studio. It supports a range of quantization levels denoted with names like Q4 K M , Q5 K S , Q8 0 , etc. — the letter/number combo encodes the bit-width and the specific quantization scheme used for that block. This is the go-to format if you're doing CPU or hybrid CPU/GPU inference, or running models locally through tools like LM Studio. A rough mental model for GGUF naming: Q2 – Q3 : very small, noticeably worse quality, mainly for extreme memory constraints Q4 K M : the most common "sweet spot" — solid quality-to-size ratio Q5 – Q6 : closer to original quality, bigger files Q8 0 : barely distinguishable from FP16 in most cases, but with much less compression benefitbitsandbytes is a library that plugs directly into Hugging Face's transformers , offering on-the-fly quantization both 8-bit via LLM.int8 and 4-bit via NF4 — a data type designed for normally-distributed weights, which is what most neural network weights look like . It's less about producing a standalone quantized file and more about loading a full-precision model with quantization applied at load time, which makes it convenient for training and fine-tuning workflows it's the backbone of QLoRA, for instance . | Method | Best For | Format | Notes | |---|---|---|---| | GPTQ | GPU inference | Pre-quantized checkpoint | Mature, widely supported | | AWQ | GPU inference | Pre-quantized checkpoint | Often better quality at same bit-width than GPTQ | | GGUF | CPU/local/hybrid inference | .gguf file | Powers llama.cpp, Ollama, LM Studio | | bitsandbytes | Fine-tuning, quick experiments | Runtime quantization | Backbone of QLoRA | A few rules of thumb that tend to hold up: Q4 K M in GGUF or standard 4-bit GPTQ/AWQ are safe defaults. transformers /vLLM on a GPU, GPTQ or AWQ checkpoints are usually the smoother path. If you're running locally via llama.cpp, Ollama, or LM Studio, GGUF is the native format.Here's what loading a 4-bit quantized model looks like with bitsandbytes through transformers : python from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig import torch quant config = BitsAndBytesConfig load in 4bit=True, bnb 4bit quant type="nf4", bnb 4bit compute dtype=torch.bfloat16, bnb 4bit use double quant=True, model = AutoModelForCausalLM.from pretrained "your-model-id", quantization config=quant config, device map="auto", tokenizer = AutoTokenizer.from pretrained "your-model-id" And if you're running locally through something like LM Studio, you typically just download a GGUF variant of the model e.g., Q4 K M and point your app at the local inference server — no quantization code required on your end, since it's baked into the file you downloaded. Quantization isn't a single technique so much as a family of trade-offs between size, speed, and quality. The good news is that for most practical applications — local assistants, RAG pipelines, prototyping — 4-bit quantization has gotten good enough that the quality hit is barely noticeable, while the accessibility gain is enormous. If you're building something that needs to run outside a well-funded GPU cluster, it's less a question of whether to quantize and more a question of which method fits your deployment target.