If you've ever tried to run a large language model on your own hardware, you've probably hit the same wall: the model is huge, your GPU's VRAM is not, and suddenly a 7B parameter model that "should" fit doesn't. This is where quantization comes in — and it's one of the most impactful techniques for making LLMs actually usable outside of a data center.
This post breaks down what quantization actually does, the major approaches you'll run into, and how to think about the trade-offs when picking one for your project.
Every parameter in a neural network is a number, and by default those numbers are usually stored as 32-bit or 16-bit floating point values (FP32 or FP16/BF16). Quantization is the process of representing those same numbers with fewer bits — commonly 8-bit, 4-bit, or even lower.
Fewer bits per parameter means:
The catch: reducing precision means reducing the information each number carries, which introduces error. The whole game in quantization research is minimizing that error while maximizing the compression.
You might think: "just round every weight to the nearest 4-bit value and call it a day." In practice, this tanks model quality fast, for a few reasons:
This is why modern quantization methods aren't just "round the numbers" — they're calibrated, often using a small dataset to figure out how to quantize with minimal damage.
GPTQ (Generative Pre-trained Transformer Quantization) is a post-training quantization method that processes weights layer-by-layer, using second-order information (an approximation of the Hessian) to figure out how to round each weight while compensating for the error introduced by previous roundings. It's calibrated against a small dataset and is one of the more established 4-bit quantization methods, especially popular for GPU inference.
AWQ takes a different angle: instead of trying to quantize every weight equally well, it identifies which weights are most important based on the activations that flow through them during inference, and protects those specifically (often by scaling them before quantization). The insight is that a small percentage of weights matter disproportionately for output quality, and preserving those gives you most of the quality of full precision at a fraction of the size.
GGUF is a file format (successor to GGML) commonly used with llama.cpp and tools built on it, like Ollama and LM Studio. It supports a range of quantization levels denoted with names like Q4_K_M
, Q5_K_S
, Q8_0
, etc. — the letter/number combo encodes the bit-width and the specific quantization scheme used for that block. This is the go-to format if you're doing CPU or hybrid CPU/GPU inference, or running models locally through tools like LM Studio.
A rough mental model for GGUF naming:
Q2
–Q3
: very small, noticeably worse quality, mainly for extreme memory constraintsQ4_K_M
: the most common "sweet spot" — solid quality-to-size ratioQ5
–Q6
: closer to original quality, bigger filesQ8_0
: barely distinguishable from FP16 in most cases, but with much less compression benefitbitsandbytes is a library that plugs directly into Hugging Face's transformers
, offering on-the-fly quantization (both 8-bit via LLM.int8() and 4-bit via NF4 — a data type designed for normally-distributed weights, which is what most neural network weights look like). It's less about producing a standalone quantized file and more about a full-precision model with quantization applied at load time, which makes it convenient for training and fine-tuning workflows (it's the backbone of QLoRA, for instance).
| Method | Best For | Format | Notes |
|---|---|---|---|
| GPTQ | GPU inference | Pre-quantized checkpoint | Mature, widely supported |
| AWQ | GPU inference | Pre-quantized checkpoint | Often better quality at same bit-width than GPTQ |
| GGUF | CPU/local/hybrid inference | ||
.gguf file |
|||
| Powers llama.cpp, Ollama, LM Studio | |||
| bitsandbytes | Fine-tuning, quick experiments | Runtime quantization | Backbone of QLoRA |
A few rules of thumb that tend to hold up:
Q4_K_M
in GGUF or standard 4-bit GPTQ/AWQ are safe defaults.transformers
/vLLM on a GPU, GPTQ or AWQ checkpoints are usually the smoother path. If you're running locally via llama.cpp, Ollama, or LM Studio, GGUF is the native format.Here's what a 4-bit quantized model looks like with bitsandbytes through transformers
:
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch
quant_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
"your-model-id",
quantization_config=quant_config,
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("your-model-id")
And if you're running locally through something like LM Studio, you typically just download a GGUF variant of the model (e.g., Q4_K_M
) and point your app at the local inference server — no quantization code required on your end, since it's baked into the file you downloaded.
Quantization isn't a single technique so much as a family of trade-offs between size, speed, and quality. The good news is that for most practical applications — local assistants, RAG pipelines, prototyping — 4-bit quantization has gotten good enough that the quality hit is barely noticeable, while the accessibility gain is enormous. If you're building something that needs to run outside a well-funded GPU cluster, it's less a question of whether to quantize and more a question of which method fits your deployment target.