This article walks through what each technique actually does, why skipping them costs real money and real latency, and then gets hands-on with five specific methods people are running in production right now.
A team fine-tunes a model for three weeks, gets the evaluation numbers they wanted, and then tries to actually serve it. The checkpoint alone is 140GB. That single number rules out almost every GPU a normal company has sitting in a rack, forces a rewrite of the deployment plan, and turns what should have been a launch week into a scramble for four A100s nobody budgeted for.
That moment is more common than it should be, and it's almost always avoidable. The model didn't need to ship at full precision with every parameter intact. It needed to ship as the leanest version of itself that still does the job, and the two techniques that get you there — quantization and pruning — are neither exotic nor new. They're just underused by teams who assume "make it smaller" means "make it worse."
This article walks through what each technique actually does, why skipping them costs real money and real latency, and then gets hands-on with five specific methods people are running in production right now, each with working code you can adapt today.
What Quantization and Pruning Actually Are #
These two get lumped together constantly, and it's worth separating them cleanly before going any further, because they solve different problems in different ways.
Quantization lowers the precision of the numbers a model is made of. A weight stored as a 16-bit floating-point number, something like 0.0023847, gets rounded and re-represented using fewer bits — an 8-bit integer or a 4-bit integer. The number of parameters in the model doesn't change at all. Every weight that existed before still exists. It just takes up less space and computes faster, the same way a high-resolution photo saved at a lower bit depth still shows every object in the frame, just with less precision in the shading.
Pruning removes weights, or entire structures, outright. A connection between two neurons, an attention head, sometimes a full layer, gets deleted because the model turns out not to need it. The parameter count itself goes down. This is closer to editing a long document by actually cutting sentences that weren't adding anything, rather than just writing everything in smaller font.
Both techniques shrink a model. They just shrink it along different axes, and as you'll see later in this article, they stack cleanly on top of each other rather than competing for the same job.
Why This Matters Right Now #
The scale problem underneath all of this is easy to understate until you see the actual numbers. A 70 billion parameter model stored in FP16 needs around 140GB of VRAM just to load, which in practice means four A100 GPUs before a single request gets served, according to ** Pristren's breakdown of LLM compression costs**. That's roughly $80,000 to $100,000 of hardware sitting idle before the model does anything useful.
Quantization changes that math directly. Compress the same 70B model to 4-bit using AWQ or GPTQ, and it drops to somewhere around 35 to 40GB — small enough to fit on a single high-end workstation card instead of a small cluster, as ** Fungies' 2026 quantization guide** lays out. Far from a marginal optimization, this is actually the difference between a model that needs a data center and one that runs on hardware a single engineer can have under their desk.
This isn't a niche concern restricted to hobbyists trying to run models locally, either. It's shaping how the biggest labs ship models in 2026. Google's Gemma 3 took its 27B model from 54GB down to roughly 14GB at 4-bit while cutting the quality loss against plain post-training quantization roughly in half, and its successor, Gemma 4, went further still, shipping quantization-aware checkpoints that get the smallest 2B variant down to about 1GB — small enough to run entirely on a phone — according to ** TensorFoundry's field guide to 2026 quantization**. Apple's on-device models on current iPhones use the same trick, squeezing weights down to 2 bits through quantization-aware training rather than guessing at scales after the fact.
The practical gains: fewer GPUs to buy or rent, lower latency per request since less data has to move through memory, and the ability to put real capability on hardware that was never going to hold the full-size model in the first place.
What Happens If You Skip This, or Do It Badly #
The flip side is worth covering honestly, because both directions of failure show up constantly in practice.
Skip compression entirely, and the failure is usually simple and expensive: a model too large to deploy on the hardware you actually have, an inference bill that makes the product commercially unviable, or latency high enough to break any use case that needs a fast response — a live chat interface, a voice assistant, an autocomplete tool. None of that is hypothetical. It's the default outcome for any team that trains a large model and assumes serving it will be someone else's problem to figure out later.
The opposite failure is quieter and more dangerous, because it doesn't announce itself the way an out-of-memory error does. Quantize too aggressively, without a proper calibration dataset, or ignore the small number of outlier weights that carry a disproportionate amount of a model's actual capability, and accuracy degrades in ways that don't always show up in a quick smoke test. ** Red Hat's own study** covering more than 500,000 evaluations of quantized models found that quality loss varies significantly by model, task, and method — some models tolerate aggressive compression fine, others fall apart fast, and the only way to know which you're dealing with is to actually benchmark the compressed version on tasks that resemble what it'll be used for, not just check that it still produces grammatical sentences. Prune carelessly, and the same pattern shows up: research on plain magnitude pruning, the simplest possible approach, found it fails dramatically on large language models (LLMs) even at fairly modest sparsity levels, as the
. LLMs turn out to be substantially harder to prune safely than the smaller networks that magnitude pruning was originally designed for.
team behind the Wanda pruning method documented directlyThe five methods in this article exist specifically to sit in the middle of those two failure modes: real, meaningful compression, done carefully enough that it doesn't quietly wreck the model you spent weeks building.
The Five Methods, at a Glance #
Before going deep on each one, here's the map. Three are quantization methods, two are pruning methods, and they differ meaningfully in how much setup they need and what they're actually optimized for.
Method | Category | Typical size reduction | Retraining needed | Best fit | |---|---|---|---|---| | bitsandbytes (NF4) | Quantization | ~4x | No (supports optional fine-tuning via QLoRA) | Fast setup, and the only option here that also enables fine-tuning | | GPTQ | Quantization | ~4x | No, calibration only | Mature GPU serving, wide pre-quantized model availability | | AWQ | Quantization | ~4x | No, calibration only | Production GPU serving, best quality-to-speed ratio on modern kernels | | SparseGPT | Pruning | ~2x (at 50% sparsity) | No, one-shot with weight update | Large models, structured 2:4 sparsity for real hardware speedups | | Wanda | Pruning | ~2x (at 50% sparsity) | No, single forward pass | Very large models where pruning speed itself matters |
Method 1: bitsandbytes (NF4 4-Bit Quantization) #
This is the method most teams should reach for first, and it's a little undersold in a lot of guides precisely because it's simple enough to use in a single function call. It's built around a data type called NF4 — NormalFloat4 — designed specifically around the fact that neural network weights tend to follow a roughly normal distribution rather than being spread evenly across the number line, so the available 4-bit values are placed where the actual weights cluster instead of being spaced out uniformly.
It's also the one method on this list that supports ** QLoRA**, meaning you can load a model in 4-bit and still fine-tune it by training small low-rank adapter weights on top, without ever touching the frozen 4-bit base weights directly. If fine-tuning is anywhere in your plan, this is the natural starting point.
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch
model_id = "meta-llama/Llama-3.1-8B-Instruct"
bnb_config = BitsAndBytesConfig(
load_in_4bit=True, # load weights in 4-bit instead of 16-bit
bnb_4bit_quant_type="nf4", # NormalFloat4: a data type tuned for
bnb_4bit_compute_dtype=torch.bfloat16, # matmuls are upcast to bfloat16 at
bnb_4bit_use_double_quant=True, # quantizes the quantization constants
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=bnb_config,
device_map="auto", # spreads layers across available
)
inputs = tokenizer("Explain quantization in one sentence.", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Walking through what actually matters here: load_in_4bit=True
is the switch that triggers the whole process, converting every linear layer's weights to 4-bit on load rather than requiring a separate offline quantization pass first, which is exactly why this is the fastest method to get running. bnb_4bit_quant_type="nf4"
picks the distribution-aware format over plain 4-bit integers, which is what keeps quality close to the original model instead of just rounding blindly.
bnb_4bit_compute_dtype=torch.bfloat16
matters because the weights sit in memory at 4-bit but get temporarily upcast to bfloat16 during the actual matrix multiplication, since GPUs don't have native 4-bit compute kernels for this yet, so this line controls that intermediate precision. And bnb_4bit_use_double_quant=True
is a small but genuinely free win: it quantizes the scaling constants used to quantize the weights in the first place, squeezing out a bit more memory with no meaningful accuracy cost.
Method 2: GPTQ (Calibrated Post-Training Quantization) #
GPTQ was one of the first 4-bit methods that actually held up well on large models, introduced in the ** original GPTQ paper** from Frantar and colleagues in 2022. The mechanism is what separates it from naive rounding: it quantizes a model layer by layer, and within each layer, it uses second-order information — an approximation of the Hessian matrix — to figure out how rounding one weight affects the ideal values of the weights around it, then adjusts the remaining unquantized weights in that layer to compensate for the error just introduced. It's error correction built directly into the quantization process, rather than quantizing every weight independently and hoping the errors don't compound.
That mechanism needs a calibration dataset, typically a few hundred samples of representative text, to estimate those Hessian statistics accurately.
from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig
import torch
model_id = "meta-llama/Llama-3.1-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
gptq_config = GPTQConfig(
bits=4, # target bit-width per weight
dataset="c4", # calibration text used to estimate the
tokenizer=tokenizer,
group_size=128, # weights are quantized in groups of 128,
desc_act=False, # skips activation-order permutation for
)
model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=gptq_config,
device_map="auto",
torch_dtype=torch.float16,
)
model.save_pretrained("./llama-3.1-8b-gptq-int4")
tokenizer.save_pretrained("./llama-3.1-8b-gptq-int4")
The dataset="c4"
line is doing the real work in this whole snippet: it's what the model runs forward passes on to collect the activation statistics GPTQ needs to compute its layer-wise error correction, and using a dataset that resembles your actual traffic tends to produce better real-world results than a generic one. group_size=128
controls the granularity of quantization — smaller groups mean more scaling constants stored (slightly more memory) in exchange for tighter accuracy, and 128 is the community-standard middle ground. desc_act=False
disables a reordering step that processes the most impactful weight columns first, which improves accuracy marginally but slows down both quantization and, in some serving setups, inference itself, so it's commonly turned off for GPU-serving-first setups where quantization is a one-time cost but inference speed happens on every request.
It's worth being upfront about GPTQ's real limitation, rather than just praising it: a January 2026 benchmark from Jarvis Labs running all four major 4-bit formats side by side on the same hardware found GPTQ trailing specifically on code generation tasks, scoring around 46% on HumanEval against AWQ and GGUF both landing near 51.8%, as reported in ** The AI Engineer's format comparison**. The likely cause is that GPTQ's column-by-column error propagation compounds more over the course of a long matrix, which hurts multi-step reasoning tasks like writing correct code more than it hurts simple next-token prediction. GPTQ remains a solid, mature, widely supported choice, especially if you already have a GPTQ checkpoint working well. It's just no longer the automatic first pick for a new setup in 2026.
Method 3: AWQ (Activation-Aware Weight Quantization) #
AWQ, introduced in ** Lin and colleagues' 2023 paper**, takes a different angle on the same underlying problem. Instead of correcting for error after the fact the way GPTQ does, it starts from an observation about which weights actually matter: by watching activations during a short calibration pass, it identifies a small percentage of "salient" weight channels — the ones that consistently produce the largest activation magnitudes and therefore have an outsized effect on the model's output. Those salient weights get protected with a scaling trick that preserves their effective precision, while everything else gets quantized aggressively.
That targeted protection is a big part of why AWQ has become the default choice for production GPU serving in 2026, particularly for instruction-tuned models where a small number of weights carrying real semantic weight can make an outsized difference to output quality.
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_path = "meta-llama/Llama-3.1-8B-Instruct"
quant_path = "llama-3.1-8b-awq"
quant_config = {
"zero_point": True, # asymmetric quantization: shifts the zero point
"q_group_size": 128, # same grouping idea as GPTQ, 128 weights per group
"w_bit": 4, # 4-bit weights
"version": "GEMM", # kernel variant tuned for batched GPU inference
}
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
zero_point=True
allows the quantized range to shift instead of forcing it to sit symmetrically around zero, which matters because real weight distributions are rarely perfectly centered, and asymmetric quantization captures that shape more faithfully. q_group_size=128
plays the identical role it does in GPTQ, controlling the accuracy-versus-memory tradeoff at the group level. w_bit=4
is the target precision. And version="GEMM"
selects the kernel AWQ compiles against at inference time; GEMM is the variant built for the batched matrix multiplications that happen when a server is handling multiple concurrent requests, which is the exact scenario production serving actually looks like.
The numbers back up why this has become the go-to: with the Marlin inference kernel, AWQ runs about 1.6x faster than the original FP16 model while retaining roughly 92% of code generation accuracy, according to ** Premai's 2026 quantization comparison**. Worth noting honestly: without an optimized kernel behind it, AWQ can actually run slower than plain FP16, so the format and the serving stack it runs on need to be chosen together, not separately.
Method 4: SparseGPT (One-Shot Structured Pruning) #
This is where the article shifts from shrinking numbers to removing weights entirely. SparseGPT, from ** Frantar and Alistarh's 2023 paper**, was the method that first proved LLMs could be pruned aggressively without retraining, at a time when the established wisdom — based on plain magnitude pruning — was that this simply didn't work on models this size. It frames pruning as a layer-wise reconstruction problem: for each layer, it decides which weights to remove and, in that same pass, updates the surviving weights in that layer to compensate for the ones just deleted, using second-order Hessian information similar in spirit to GPTQ's approach.
The practical detail that matters most here is the sparsity pattern. Unstructured sparsity — zeroing out whatever individual weights score lowest with no pattern to where they sit — saves memory on disk but doesn't actually speed anything up on standard GPU hardware, because the hardware still has to load every weight from memory regardless of whether it's zero. NVIDIA's 2:4 structured sparsity pattern, exactly two zeros in every group of four consecutive weights, is what changes that: Sparse Tensor Cores on Ampere, Hopper, and Blackwell GPUs can skip the zeroed weights during matrix multiplication entirely, delivering a real, measurable speedup rather than just a smaller file, as explained in ** Spheron's guide to running SparseGPT and Wanda on GPU cloud hardware**.
git clone https://github.com/IST-DASLab/sparsegpt
cd sparsegpt
python llama.py meta-llama/Llama-3.1-8B-Instruct c4 \
--sparsity 0.5 \ # target: 50% of weights removed overall
--prunen 2 --prunem 4 \ # enforce a 2:4 pattern, 2 zeros in every group of 4,
--save llama-3.1-8b-sparsegpt-2-4
The two positional arguments — the model identifier and c4
— tell the script which model to prune and which calibration dataset to run forward passes on to estimate the Hessian statistics the pruning decisions are based on, functionally the same role calibration data plays for GPTQ. --sparsity 0.5
sets the overall target; half the weights across pruned layers get removed. --prunen 2 --prunem 4
is the flag pair that actually enforces the 2:4 structured pattern rather than leaving the pruning unstructured, and it's the single most important setting in this command if the goal is real inference speedup rather than just a smaller checkpoint on disk. Expect this to take somewhere in the range of an hour on a single H100 for a 70B model, considerably less for something in the 7B to 8B range.
Method 5: Wanda (Pruning by Weights and Activations) #
Wanda, short for Pruning by Weights and Activations, from ** Sun and colleagues' 2023 paper**, takes SparseGPT's core insight and strips it down to something much simpler. Instead of solving a full layer-wise reconstruction problem with Hessian inversion, Wanda scores each weight using just the product of its magnitude and the L2 norm of its corresponding input activation — a metric that can be computed in a single forward pass through the model. There's no weight update step afterwards at all; the surviving weights are simply left exactly as they were.
That simplicity translates directly into speed. Because there's no Hessian to invert and no iterative column-by-column solving, Wanda's own paper reports it can be roughly 300 times faster to compute than SparseGPT, and separate benchmarking on 70B-class models found it runs 5 to 10 times faster in wall-clock terms with roughly half the peak memory, according to ** Spheron's practical comparison**. Quality-wise, the comparison isn't a clean win for either method across the board. SparseGPT tends to edge out Wanda on smaller models around the 7B mark under 2:4 structured sparsity, while Wanda holds up better on larger models like LLaMA-30B, per the original paper's own reported results.
git clone https://github.com/locuslab/wanda
cd wanda
python main.py \
--model meta-llama/Llama-3.1-8B-Instruct \
--prune_method wanda \ # selects the magnitude-times-activation metric
--sparsity_ratio 0.5 \ # remove 50% of weights overall
--sparsity_type 2:4 \ # structured pattern for real GPU speedups
--save out/llama-3.1-8b-wanda-2-4
--prune_method wanda
is what selects this specific scoring approach over the script's other supported methods, including plain magnitude pruning and SparseGPT itself, since the two are often implemented side by side in the same tooling for direct comparison. --sparsity_ratio
and --sparsity_type
mirror SparseGPT's flags almost exactly, with half the weights removed and structured into the 2:4 pattern hardware can actually exploit. The practical reason to reach for Wanda specifically over SparseGPT is when the model is large enough — or the calibration set numerous enough — that SparseGPT's inverse Hessian computation becomes the bottleneck in your workflow rather than the pruning decision itself.
Stacking Them: Pruning and Quantization Together #
These five methods aren't a menu where you pick exactly one. Pruning and quantization attack different parts of the same problem, so they combine directly, and the combined result is bigger than either technique alone. Take a 70B model, prune it first with SparseGPT or Wanda down to 50% structured sparsity, then quantize what's left with AWQ or GPTQ, and a model that needed 140GB in its original FP16 form can land around 17 to 18GB — small enough to run comfortably on a single high-end consumer GPU, per ** Spheron's combined benchmarking**.
The order matters, and it's not arbitrary. Pruning first and quantizing second works because the quantization step calibrates against the model's actual final weight distribution, including the gaps pruning already introduced. Reverse the order and quantize first, then prune, and the pruning step is now making its removal decisions based on weights that have already been rounded and distorted, compounding two sources of error against each other instead of letting the second step correct cleanly for what the first one changed.
Choosing the Right Method for Your Situation #
With five real options on the table, the actual decision usually comes down to what you're optimizing for, and the comparison table from earlier maps fairly directly onto real-world choices. If fine-tuning is anywhere in the plan — not just inference — bitsandbytes with QLoRA is the only method on this list built for that from the ground up. If you're serving at scale through something like vLLM and raw throughput matters most, AWQ with the Marlin kernel is the current default for good reason. If you already have a GPTQ checkpoint working reliably in production, there's rarely a strong case to migrate purely for the sake of it, though a new project is better served starting with AWQ today. If you're deploying to a laptop, an edge device, or running through something like Ollama or LM Studio, that world runs on the GGUF format rather than any of the three quantization methods detailed above, since GGUF is built specifically for efficient CPU inference, and it's worth knowing that most compressed models eventually get converted into it for that last mile of deployment.
For pruning specifically, the choice usually comes down to model size and how much compute you're willing to spend on the pruning pass itself. SparseGPT's extra weight-update step tends to edge out Wanda's quality on smaller models in the 7B range. Wanda's dramatically lower compute cost makes it the more practical choice as models get larger, when SparseGPT's Hessian computation starts to become a real bottleneck rather than a rounding error in your timeline.
Conclusion #
None of these five methods make a model worse in any meaningful sense, done properly. They make it honest. Most large models ship with more precision and more parameters than the task in front of them actually requires, carried over from training runs optimized for a different goal than the one deployment cares about. Quantization and pruning are how you find out what a model genuinely needs to keep doing its job well, and cut the rest.
Start with whichever of these five fits the constraint you're actually up against right now — memory, latency, hardware you don't have, or a fine-tuning step you still need to run — rather than chasing the method with the best benchmark number on a task that isn't yours. Benchmark the result on something that resembles your real traffic before you trust it. That's the whole discipline here, and it's a lot more approachable than the size of these models makes it feel.
is a software engineer and technical writer passionate about leveraging cutting-edge technologies to craft compelling narratives, with a keen eye for detail and a knack for simplifying complex concepts. You can also find Shittu on