# Shrink Your LLM by 75% and (Mostly) Keep Its Brain: Quantization Explained

> Source: <https://dev.to/pollab_d/shrink-your-llm-by-75-and-mostly-keep-its-brain-quantization-explained-4kgn>
> Published: 2026-07-09 07:02:01+00:00

If you've ever tried to run a large language model on your own hardware, you've probably hit the same wall: the model is *huge*, your GPU's VRAM is not, and suddenly a 7B parameter model that "should" fit doesn't. This is where quantization comes in — and it's one of the most impactful techniques for making LLMs actually usable outside of a data center.

This post breaks down what quantization actually does, the major approaches you'll run into, and how to think about the trade-offs when picking one for your project.

Every parameter in a neural network is a number, and by default those numbers are usually stored as 32-bit or 16-bit floating point values (FP32 or FP16/BF16). Quantization is the process of representing those same numbers with fewer bits — commonly 8-bit, 4-bit, or even lower.

Fewer bits per parameter means:

The catch: reducing precision means reducing the information each number carries, which introduces error. The whole game in quantization research is minimizing that error while maximizing the compression.

You might think: "just round every weight to the nearest 4-bit value and call it a day." In practice, this tanks model quality fast, for a few reasons:

This is why modern quantization methods aren't just "round the numbers" — they're calibrated, often using a small dataset to figure out *how* to quantize with minimal damage.

GPTQ (Generative Pre-trained Transformer Quantization) is a post-training quantization method that processes weights layer-by-layer, using second-order information (an approximation of the Hessian) to figure out how to round each weight while compensating for the error introduced by previous roundings. It's calibrated against a small dataset and is one of the more established 4-bit quantization methods, especially popular for GPU inference.

AWQ takes a different angle: instead of trying to quantize every weight equally well, it identifies which weights are most important based on the *activations* that flow through them during inference, and protects those specifically (often by scaling them before quantization). The insight is that a small percentage of weights matter disproportionately for output quality, and preserving those gives you most of the quality of full precision at a fraction of the size.

GGUF is a file format (successor to GGML) commonly used with llama.cpp and tools built on it, like Ollama and LM Studio. It supports a range of quantization levels denoted with names like `Q4_K_M`

, `Q5_K_S`

, `Q8_0`

, etc. — the letter/number combo encodes the bit-width and the specific quantization scheme used for that block. This is the go-to format if you're doing CPU or hybrid CPU/GPU inference, or running models locally through tools like LM Studio.

A rough mental model for GGUF naming:

`Q2`

–`Q3`

: very small, noticeably worse quality, mainly for extreme memory constraints`Q4_K_M`

: the most common "sweet spot" — solid quality-to-size ratio`Q5`

–`Q6`

: closer to original quality, bigger files`Q8_0`

: barely distinguishable from FP16 in most cases, but with much less compression benefitbitsandbytes is a library that plugs directly into Hugging Face's `transformers`

, offering on-the-fly quantization (both 8-bit via LLM.int8() and 4-bit via NF4 — a data type designed for normally-distributed weights, which is what most neural network weights look like). It's less about producing a standalone quantized file and more about loading a full-precision model with quantization applied at load time, which makes it convenient for training and fine-tuning workflows (it's the backbone of QLoRA, for instance).

| Method | Best For | Format | Notes |
|---|---|---|---|
| GPTQ | GPU inference | Pre-quantized checkpoint | Mature, widely supported |
| AWQ | GPU inference | Pre-quantized checkpoint | Often better quality at same bit-width than GPTQ |
| GGUF | CPU/local/hybrid inference |
`.gguf` file |
Powers llama.cpp, Ollama, LM Studio |
| bitsandbytes | Fine-tuning, quick experiments | Runtime quantization | Backbone of QLoRA |

A few rules of thumb that tend to hold up:

`Q4_K_M`

in GGUF or standard 4-bit GPTQ/AWQ are safe defaults.`transformers`

/vLLM on a GPU, GPTQ or AWQ checkpoints are usually the smoother path. If you're running locally via llama.cpp, Ollama, or LM Studio, GGUF is the native format.Here's what loading a 4-bit quantized model looks like with bitsandbytes through `transformers`

:

``` python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "your-model-id",
    quantization_config=quant_config,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("your-model-id")
```

And if you're running locally through something like LM Studio, you typically just download a GGUF variant of the model (e.g., `Q4_K_M`

) and point your app at the local inference server — no quantization code required on your end, since it's baked into the file you downloaded.

Quantization isn't a single technique so much as a family of trade-offs between size, speed, and quality. The good news is that for most practical applications — local assistants, RAG pipelines, prototyping — 4-bit quantization has gotten good enough that the quality hit is barely noticeable, while the accessibility gain is enormous. If you're building something that needs to run outside a well-funded GPU cluster, it's less a question of *whether* to quantize and more a question of *which* method fits your deployment target.