Weight-Only Quantization

A technical tutorial published on Machine Learning Mastery details weight-only quantization, a technique that stores transformer weights as integers while keeping activations in FP16 or BF16 to cut the memory traffic that dominates autoregressive decoding at small batch sizes. The post walks through asymmetric affine quantization with scale and zero point, post-training quantization in PyTorch using TorchAO (installed via `python -m pip install torchao transformers accelerate`), GPTQ and AWQ, GGUF conversion through llama.cpp, and how to evaluate the memory, speed, and quality effects. It notes an INT8 weight uses half the raw storage of an FP16 or BF16 weight, and that low-precision matrix speedups depend on hardware, kernels, and model shapes.

During autoregressive decoding, a transformer reads its weights for every new token. At small batch sizes, moving these weights can take more time than computation. Quantization reduces this cost by storing weights with fewer bits. For example, an INT8 weight uses half the raw storage of an FP16 or BF16 weight. Quantization can also make computation faster. Many AI GPUs can perform low-precision matrix operations faster than FP16 operations. The actual speedup depends on the hardware, kernels, and model shapes. Weight-only quantization stores selected weights as integers while keeping activations in FP16 or BF16. This post begins with the asymmetric affine mapping, then shows how to quantize and run models with PyTorch and other inference tools. In this post, you will learn about: - How weight-only quantization stores and reconstructs model weights - How post-training methods and GGUF conversion produce deployable artifacts - How to check the memory, speed, and quality effects of quantization Let’s get started. Overview This post is divided into five parts; they are: - Asymmetric Affine Quantization - Post-Training Quantization with PyTorch - GPTQ and AWQ - Quantizing a Model to GGUF - Evaluating Weight-Only Quantization Note: The code examples in this post use TorchAO and require a CUDA-enabled PyTorch installation. You can install the dependencies with: python -m pip install torchao transformers accelerate The GPTQ and AWQ sections have their own installation steps. GGUF conversion requires a built llama.cpp checkout. Asymmetric Affine Quantization Integer quantization maps a range of real values onto a finite set of integer codes. An asymmetric affine quantizer uses a scale $s$ and a zero point $z$: $$ q = \operatorname{clamp}\Big \operatorname{round}\big \frac{x}{s}\big + z, q {\min}, q {\max}\Big $$ Dequantization reconstructs an approximate floating-point value: $$ \hat{x} = s q-z $$ For an unsigned $b$-bit range, $q {\min}=0$ and $q {\max}=2^b-1$. Given the minimum and maximum values to represent, one common choice is: $$ \begin{aligned} s &= \frac{x {\max}-x {\min}}{q {\max}-q {\min}} \\ z &= \operatorname{clamp}\Big \operatorname{round}\big q {\min}-\frac{x {\min}}{s}\big ,q {\min},q {\max}\Big \end{aligned} $$ The scale determines the distance between adjacent representable values. The zero point is the integer code for real zero. Including zero in the real range ensures that it can be represented exactly. Quantize and reconstruct a tensor to show the error from integer rounding. import torch def quantize asymmetric x, bits=8 : if not 1 <= bits <= 8: raise ValueError "this uint8 example supports 1 to 8 bits" if not x.is floating point : raise TypeError "x must be floating point" max and min of quantized unsigned int values qmin = 0 qmax = 2 bits - 1 max and min of actual float values xmin = torch.minimum x.amin , x.new zeros xmax = torch.maximum x.amax , x.new zeros compute the scale and zero point, then quantize the actual values scale = xmax - xmin / qmax - qmin scale = scale.clamp min torch.finfo x.dtype .eps zero point = torch.round qmin - xmin / scale .clamp qmin, qmax q = torch.round x / scale + zero point .clamp qmin, qmax return q.to torch.uint8 , scale, zero point.to torch.int32 def dequantize q, scale, zero point : return scale q.float - zero point.float x = torch.tensor -2.3, -0.2, 0.0, 0.7, 3.1 q, scale, zero point = quantize asymmetric x x hat = dequantize q, scale, zero point print "scale:", scale.item print "zero point:", zero point.item print "quantized:", q print "maximum error:", x - x hat .abs .max .item 12345678910111213141516171819202122232425262728293031323334 import torch def quantize asymmetric x, bits=8 : if not 1 <= bits <= 8: raise ValueError "this uint8 example supports 1 to 8 bits" if not x.is floating point : raise TypeError "x must be floating point" max and min of quantized unsigned int values qmin = 0 qmax = 2 bits - 1 max and min of actual float values xmin = torch.minimum x.amin , x.new zeros xmax = torch.maximum x.amax , x.new zeros compute the scale and zero point, then quantize the actual values scale = xmax - xmin / qmax - qmin scale = scale.clamp min torch.finfo x.dtype .eps zero point = torch.round qmin - xmin / scale .clamp qmin, qmax q = torch.round x / scale + zero point .clamp qmin, qmax return q.to torch.uint8 , scale, zero point.to torch.int32 def dequantize q, scale, zero point : return scale q.float - zero point.float x = torch.tensor -2.3, -0.2, 0.0, 0.7, 3.1 q, scale, zero point = quantize asymmetric x x hat = dequantize q, scale, zero point print "scale:", scale.item print "zero point:", zero point.item print "quantized:", q print "maximum error:", x - x hat .abs .max .item The example demonstrates the numerical mapping, not efficient storage. A uint8 tensor always uses one byte per element, even if bits is less than eight. A deployment format must pack sub-byte values to obtain the expected storage reduction. This example uses one scale and zero point for the whole tensor. Per-channel and per-group quantization often preserve LLM weight quality better because each channel or group gets its own range. For a linear layer $\mathbf{y}=\mathbf{x}\mathbf{W}^\top$, weight-only quantization stores an integer matrix $\mathbf{Q}$ plus scales and zero points instead of $\mathbf{W}$: $$ \mathbf{y} \approx \mathbf{x}\big s \mathbf{Q}-z \big ^\top $$ The input remains floating point. An efficient kernel would use the quantized weights directly instead of reconstructing the floating-point copy of $\mathbf{W}$ like the equation shows. The asymmetric affine mapping is the explanatory model used in this post. Production kernels and checkpoint formats may use a fixed zero point or another encoding. The runtime must support the exact quantization scheme and packed layout stored in the checkpoint. Post-Training Quantization with PyTorch PyTorch’s current quantization development is centered in TorchAO. Its quantize function replaces the weights of matching torch.nn.Linear modules in place. Applying it to an evaluated, trained model is post-training quantization: no training step is performed. The following example requests asymmetric, per-channel INT8 weight-only quantization. The activations remain BF16: python Apply per-channel INT8 weight-only quantization with TorchAO. import torch import torch.nn as nn from torchao.quantization import IntxWeightOnlyConfig, MappingType, PerAxis, quantize , if not torch.cuda.is available : raise RuntimeError "This example requires a CUDA GPU" model = nn.Sequential nn.Linear 2048, 2048, bias=False , nn.ReLU , nn.Linear 2048, 2048, bias=False , .eval .to device="cuda", dtype=torch.bfloat16 config = IntxWeightOnlyConfig weight dtype=torch.int8, granularity=PerAxis 0 , mapping type=MappingType.ASYMMETRIC, quantize model, config x = torch.randn 1, 2048, device="cuda", dtype=torch.bfloat16 with torch.inference mode : output = model x print output.shape 123456789101112131415161718192021222324252627282930 import torchimport torch.nn as nnfrom torchao.quantization import IntxWeightOnlyConfig, MappingType, PerAxis, quantize , if not torch.cuda.is available : raise RuntimeError "This example requires a CUDA GPU" model = nn.Sequential nn.Linear 2048, 2048, bias=False , nn.ReLU , nn.Linear 2048, 2048, bias=False , .eval .to device="cuda", dtype=torch.bfloat16 config = IntxWeightOnlyConfig weight dtype=torch.int8, granularity=PerAxis 0 , mapping type=MappingType.ASYMMETRIC, quantize model, config x = torch.randn 1, 2048, device="cuda", dtype=torch.bfloat16 with torch.inference mode : output = model x print output.shape PerAxis 0 assigns separate quantization parameters to the rows of each linear weight matrix. IntxWeightOnlyConfig also supports per-group granularity. Its available packing formats and optimized kernels depend on the TorchAO version and target device. TorchAO also provides hardware-oriented presets such as Int8WeightOnlyConfig and Int4WeightOnlyConfig . These presets may use a different mapping from the explicit asymmetric example. You should use the configuration documented for the target hardware, then benchmark the complete model. A mathematically valid configuration is not necessarily the fastest one. The transformers library can apply a TorchAO configuration while loading a model. For example: python Load a causal model with a TorchAO quantization configuration. import torch from torchao.quantization import Int4WeightOnlyConfig from transformers import AutoModelForCausalLM, AutoTokenizer, TorchAoConfig model id = "Qwen/Qwen2.5-0.5B-Instruct" quantization config = TorchAoConfig quant type=Int4WeightOnlyConfig group size=128, int4 packing format="tile packed to 4d", int4 choose qparams algorithm="hqq", model = AutoModelForCausalLM.from pretrained model id, dtype=torch.bfloat16, device map="auto", quantization config=quantization config, tokenizer = AutoTokenizer.from pretrained model id inputs = tokenizer "Explain post-training quantization.", return tensors="pt" inputs = inputs.to model.device with torch.inference mode : generated = model.generate inputs, max new tokens=32 print tokenizer.decode generated 0 , skip special tokens=True 123456789101112131415161718192021222324252627 import torchfrom torchao.quantization import Int4WeightOnlyConfigfrom transformers import AutoModelForCausalLM, AutoTokenizer, TorchAoConfig model id = "Qwen/Qwen2.5-0.5B-Instruct"quantization config = TorchAoConfig quant type=Int4WeightOnlyConfig group size=128, int4 packing format="tile packed to 4d", int4 choose qparams algorithm="hqq", model = AutoModelForCausalLM.from pretrained model id, dtype=torch.bfloat16, device map="auto", quantization config=quantization config, tokenizer = AutoTokenizer.from pretrained model id inputs = tokenizer "Explain post-training quantization.", return tensors="pt" inputs = inputs.to model.device with torch.inference mode : generated = model.generate inputs, max new tokens=32 print tokenizer.decode generated 0 , skip special tokens=True This is calibration-free weight-only PTQ. The explicit HQQ parameter choice and tiled packing avoid an optional MSLK kernel dependency in TorchAO 0.17 on the tested RTX 4090. Other TorchAO versions and devices may support different layouts. Static activation quantization is a different workflow: it prepares an exported model, runs representative inputs to collect activation ranges, and then converts the model. GPTQ and AWQ Round-to-nearest quantization treats each weight locally. GPTQ and AWQ use calibration activations to reduce the effect of quantization error on model outputs. Rounding each weight independently ignores how often and how strongly its input channel contributes to outputs. Calibration examples let a quantizer estimate this effect and reduce output error, though the result depends on how representative those examples are. GPTQ quantizes weights layer by layer using approximate second-order information. As weights are quantized, it adjusts the remaining weights to compensate for earlier error. The transformers library now uses GPT-QModel as its GPTQ backend. Install the required packages to use GPTQ for quantization: Install the GPTQ conversion dependencies. python -m pip install setuptools --upgrade wheel accelerate optimum transformers python -m pip install gptqmodel --no-build-isolation 12 python -m pip install setuptools --upgrade wheel accelerate optimum transformerspython -m pip install gptqmodel --no-build-isolation Then you can quantize and save a small model. The GPTQConfig setting dataset="c4" provides input samples for calibration. Setting sym=False requests the asymmetric form. Below is an example of quantizing a small model to INT4: python Calibrate and save an INT4 GPTQ checkpoint. from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig model id = "facebook/opt-125m" tokenizer = AutoTokenizer.from pretrained model id quantization config = GPTQConfig bits=4, 2, 3, 4, or 8 group size=128, dataset="c4", tokenizer=tokenizer, sym=False, model = AutoModelForCausalLM.from pretrained model id, device map="auto", quantization config=quantization config, model.save pretrained "opt-125m-gptq" tokenizer.save pretrained "opt-125m-gptq" print "Saved the GPTQ checkpoint to opt-125m-gptq" 1234567891011121314151617181920 from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig model id = "facebook/opt-125m"tokenizer = AutoTokenizer.from pretrained model id quantization config = GPTQConfig bits=4, 2, 3, 4, or 8 group size=128, dataset="c4", tokenizer=tokenizer, sym=False, model = AutoModelForCausalLM.from pretrained model id, device map="auto", quantization config=quantization config, model.save pretrained "opt-125m-gptq" tokenizer.save pretrained "opt-125m-gptq" print "Saved the GPTQ checkpoint to opt-125m-gptq" To load the saved GPTQ checkpoint in a fresh process and run one inference: python Reload the saved GPTQ checkpoint and generate text. import torch from transformers import AutoModelForCausalLM, AutoTokenizer checkpoint = "opt-125m-gptq" tokenizer = AutoTokenizer.from pretrained checkpoint model = AutoModelForCausalLM.from pretrained checkpoint, device map="auto", model.eval inputs = tokenizer "Weight quantization makes inference", return tensors="pt" inputs = inputs.to model.device with torch.inference mode : output = model.generate inputs, max new tokens=32, pad token id=tokenizer.eos token id, print tokenizer.decode output 0 , skip special tokens=True 12345678910111213141516171819202122 import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer checkpoint = "opt-125m-gptq"tokenizer = AutoTokenizer.from pretrained checkpoint model = AutoModelForCausalLM.from pretrained checkpoint, device map="auto", model.eval inputs = tokenizer "Weight quantization makes inference", return tensors="pt" inputs = inputs.to model.device with torch.inference mode : output = model.generate inputs, max new tokens=32, pad token id=tokenizer.eos token id, print tokenizer.decode output 0 , skip special tokens=True Asymmetric GPTQ checkpoints have fewer compatible kernels than the default mapping. For example, the Marlin backend does not support asymmetric GPTQ. You should choose the quantization scheme and inference backend together. Activation-aware Weight Quantization AWQ is another quantization method. It identifies weights that are important under representative activations and searches for scaling factors that protect them before low-bit quantization. The --model path argument points to the original, unquantized model in Hugging Face format. From a checkout of the reference llm-awq implementation with its dependencies installed, run the search first and then create the real INT4 checkpoint: Install the reference implementation in a separate environment before running its commands: Install AutoAWQ and the Hugging Face Hub client. git clone https://github.com/mit-han-lab/llm-awq.git cd llm-awq python -m pip install -e . python -m pip install huggingface hub 1234 git clone https://github.com/mit-han-lab/llm-awq.gitcd llm-awqpython -m pip install -e .python -m pip install huggingface hub The reference project documents Python 3.10 for this environment. Use its installation instructions https://github.com/mit-han-lab/llm-awq install for the matching CUDA extensions. Download the model with hf download TinyLlama/TinyLlama-1.1B-Chat-v1.0 --local-dir models/tinyllama 1 hf download TinyLlama/TinyLlama-1.1B-Chat-v1.0 --local-dir models/tinyllama then run the following commands: Search for AWQ scales and quantize the TinyLlama checkpoint. Search for AWQ scaling parameters. python -m awq.entry \ --model path models/tinyllama \ --w bit 4 \ --q group size 128 \ --run awq \ --dump awq awq cache/tinyllama-w4-g128.pt Create the quantized checkpoint. python -m awq.entry \ --model path models/tinyllama \ --w bit 4 \ --q group size 128 \ --load awq awq cache/tinyllama-w4-g128.pt \ --q backend real \ --dump quant quant cache/tinyllama-w4-g128-awq-v2.pt 12345678910111213141516 Search for AWQ scaling parameters.python -m awq.entry \ --model path models/tinyllama \ --w bit 4 \ --q group size 128 \ --run awq \ --dump awq awq cache/tinyllama-w4-g128.pt Create the quantized checkpoint.python -m awq.entry \ --model path models/tinyllama \ --w bit 4 \ --q group size 128 \ --load awq awq cache/tinyllama-w4-g128.pt \ --q backend real \ --dump quant quant cache/tinyllama-w4-g128-awq-v2.pt The --run awq command obtains its calibration input internally. It uses 128 samples from the validation split of mit-han-lab/pile-val-backup and forms sequences of 512 tokens. This reference CLI does not expose an argument for selecting another text dataset. Omitting --no zero point keeps the implementation’s zero-point quantization. The raw .pt file produced by --dump quant is not a complete Hugging Face checkpoint. It contains the quantized weights but not the model configuration or tokenizer. The following script packages them together in a local directory: Package an AWQ weight file with its model configuration and tokenizer. from pathlib import Path import shutil from transformers import AutoConfig, AutoTokenizer, AwqConfig source model = "models/tinyllama" quantized weights = "quant cache/tinyllama-w4-g128-awq-v2.pt" output dir = Path "models/tinyllama-awq" output dir.mkdir parents=True, exist ok=True awq config = AwqConfig bits=4, group size=128, zero point=True, backend="llm-awq", version="gemv", config = AutoConfig.from pretrained source model config.quantization config = awq config config.save pretrained output dir tokenizer = AutoTokenizer.from pretrained source model tokenizer.save pretrained output dir shutil.copyfile quantized weights, output dir / "pytorch model.bin", 1234567891011121314151617181920212223242526272829 from pathlib import Pathimport shutil from transformers import AutoConfig, AutoTokenizer, AwqConfig source model = "models/tinyllama"quantized weights = "quant cache/tinyllama-w4-g128-awq-v2.pt"output dir = Path "models/tinyllama-awq" output dir.mkdir parents=True, exist ok=True awq config = AwqConfig bits=4, group size=128, zero point=True, backend="llm-awq", version="gemv", config = AutoConfig.from pretrained source model config.quantization config = awq configconfig.save pretrained output dir tokenizer = AutoTokenizer.from pretrained source model tokenizer.save pretrained output dir shutil.copyfile quantized weights, output dir / "pytorch model.bin", The transformers library can then identify the AWQ format from the checkpoint’s quantization config , load the quantized weights, and run inference: python Load the packaged AWQ checkpoint and generate text. import torch from transformers import AutoModelForCausalLM, AutoTokenizer model id = "models/tinyllama-awq" tokenizer = AutoTokenizer.from pretrained model id model = AutoModelForCausalLM.from pretrained model id, device map="auto", model.eval inputs = tokenizer "Explain weight quantization.", return tensors="pt" inputs = inputs.to model.device with torch.inference mode : output = model.generate inputs, max new tokens=64, pad token id=tokenizer.eos token id, print tokenizer.decode output 0 , skip special tokens=True 12345678910111213141516171819202122 import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer model id = "models/tinyllama-awq"tokenizer = AutoTokenizer.from pretrained model id model = AutoModelForCausalLM.from pretrained model id, device map="auto", model.eval inputs = tokenizer "Explain weight quantization.", return tensors="pt" inputs = inputs.to model.device with torch.inference mode : output = model.generate inputs, max new tokens=64, pad token id=tokenizer.eos token id, print tokenizer.decode output 0 , skip special tokens=True The model architecture, group size, zero-point convention, packed layout, and installed AWQ backend must be compatible. Quantizing a large model can take substantial time and memory. Before starting, check whether a compatible quantized checkpoint already exists. If you quantize it yourself, save the tokenizer and complete quantization configuration with the packed weights. Quantizing a Model to GGUF GGUF is a container format used by the llama.cpp ecosystem. It stores model metadata, tokenizer information, tensors, and their types in one or more files. GGUF supports many quantization types; it is not itself one quantization algorithm. The llama.cpp workflow has two stages. First, convert the original Hugging Face model to a high-precision GGUF file such as bfloat16 , then quantize that file. The following example uses the original, unquantized TinyLlama/TinyLlama-1.1B-Chat-v1.0 checkpoint. From a built llama.cpp checkout: Install the conversion script’s Python dependencies in a separate environment and build the binaries first. The conversion requirements may select a different PyTorch build from the one used for the CUDA examples earlier in this post. This CUDA build needs CMake, a C++ compiler, and the CUDA toolkit; omit -DGGML CUDA=ON for a CPU build. Clone and build llama.cpp with GGUF conversion support. git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp python -m pip install -r requirements/requirements-convert hf to gguf.txt python -m pip install huggingface hub cmake -B build -DGGML CUDA=ON cmake --build build --config Release hf download TinyLlama/TinyLlama-1.1B-Chat-v1.0 --local-dir models/tinyllama 1234567 git clone https://github.com/ggml-org/llama.cpp.gitcd llama.cpppython -m pip install -r requirements/requirements-convert hf to gguf.txtpython -m pip install huggingface hubcmake -B build -DGGML CUDA=ONcmake --build build --config Releasehf download TinyLlama/TinyLlama-1.1B-Chat-v1.0 --local-dir models/tinyllama Convert TinyLlama to GGUF and quantize it to Q4 K M. python3 convert hf to gguf.py models/tinyllama \ --outfile tinyllama-bf16.gguf \ --outtype bf16 ./build/bin/llama-quantize tinyllama-bf16.gguf tinyllama-Q4 K M.gguf Q4 K M 12345 python3 convert hf to gguf.py models/tinyllama \ --outfile tinyllama-bf16.gguf \ --outtype bf16 ./build/bin/llama-quantize tinyllama-bf16.gguf tinyllama-Q4 K M.gguf Q4 K M Q4 K M is a mixed GGUF quantization type that uses different encodings for different tensors. Its file size includes scales, metadata, and tensors stored at other precisions, so it is not exactly four bits per model parameter. Its name does not mean every value follows the affine formula shown earlier. You can use llama-cli to load the quantized GGUF file and generate text directly from its packed weights: Run the quantized GGUF model with llama.cpp. ./build/bin/llama-cli \ -m tinyllama-Q4 K M.gguf \ -p "Explain weight-only quantization in one paragraph." \ -n 128 --single-turn 1234 ./build/bin/llama-cli \ -m tinyllama-Q4 K M.gguf \ -p "Explain weight-only quantization in one paragraph." \ -n 128 --single-turn This confirms that llama.cpp can generate from the packed file. TinyLlama may give an inaccurate explanation, so use separate quality checks for your task. Some GGUF models are available on the Hugging Face Hub and can be run directly: Run a GGUF model directly from the Hugging Face Hub. ./build/bin/llama-cli -hf Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4 K M \ -p "What is a quantized model?" -n 64 --single-turn 12 ./build/bin/llama-cli -hf Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4 K M \ -p "What is a quantized model?" -n 64 --single-turn If you prefer to use Python code, transformers also accepts a GGUF filename for supported architectures: Install the Hugging Face library with GGUF loading support. python -m pip install --upgrade transformers gguf 1 python -m pip install --upgrade transformers gguf Use a recent transformers release for the Qwen3.5 example below. Loading may dequantize tensors and require much more memory than the GGUF file size suggests. Load a GGUF checkpoint through the Hugging Face library. from transformers import AutoModelForCausalLM, AutoTokenizer model id = "unsloth/Qwen3.5-4B-GGUF" filename = "Qwen3.5-4B-Q4 K M.gguf" model = AutoModelForCausalLM.from pretrained model id, gguf file=filename tokenizer = AutoTokenizer.from pretrained model id, gguf file=filename print type model . name , type tokenizer . name 12345678 from transformers import AutoModelForCausalLM, AutoTokenizer model id = "unsloth/Qwen3.5-4B-GGUF"filename = "Qwen3.5-4B-Q4 K M.gguf" model = AutoModelForCausalLM.from pretrained model id, gguf file=filename tokenizer = AutoTokenizer.from pretrained model id, gguf file=filename print type model . name , type tokenizer . name But you should not assume that transformers preserves compressed inference. It keeps GGUF blocks packed only for a narrow set of model architectures, devices, and quantization types; other combinations are dequantized while loading. You can use llama.cpp when broad GGUF compatibility and direct execution of packed weights are required. You should quantize from the highest-quality source available. Requantizing an already quantized model compounds approximation error, and llama-quantize warns that it can severely reduce quality. Evaluating Weight-Only Quantization Evaluate the quantized model on representative inputs. If quality degrades beyond your application’s tolerance, try a different calibration set, quantizer, or bit width. Some quantizers use calibration examples to reduce output error between the original and quantized model. That proxy does not replace application quality metrics. Useful checks include: - Perplexity on representative text - Accuracy or task-specific scores - Compare greedy generations side by side - Compare sampled generations across several seeds, prompt lengths, and temperatures After creating opt-125m-gptq with the earlier command, run this script on one CUDA GPU. It loads and measures the FP16 baseline and the GPTQ checkpoint one at a time so their peak allocated memory measurements are comparable. The timing includes generate but excludes model loading and tokenization: python Compare memory use and generation time across model checkpoints. import gc import time import torch from transformers import AutoModelForCausalLM, AutoTokenizer if not torch.cuda.is available : raise RuntimeError "This comparison needs a CUDA GPU and a GPTQ CUDA kernel" tokenizer = AutoTokenizer.from pretrained "facebook/opt-125m" inputs = tokenizer "Weight quantization makes inference", return tensors="pt" .to "cuda" def measure checkpoint path : torch.cuda.empty cache torch.cuda.reset peak memory stats model = AutoModelForCausalLM.from pretrained path, device map="cuda:0", dtype=torch.float16, .eval def generate current model : with torch.inference mode : return current model.generate inputs, max new tokens=32, do sample=False, pad token id=tokenizer.eos token id, generate model warm up torch.cuda.synchronize times = for in range 3 : start = time.perf counter ids = generate model torch.cuda.synchronize times.append time.perf counter - start result = { "median seconds": sorted times 1 , "peak allocated gib": torch.cuda.max memory allocated / 2 30, "output": tokenizer.decode ids 0 , skip special tokens=True , } del model gc.collect torch.cuda.empty cache return result for checkpoint in "facebook/opt-125m", "opt-125m-gptq" : print checkpoint, measure checkpoint checkpoint 1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950 import gcimport time import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer if not torch.cuda.is available : raise RuntimeError "This comparison needs a CUDA GPU and a GPTQ CUDA kernel" tokenizer = AutoTokenizer.from pretrained "facebook/opt-125m" inputs = tokenizer "Weight quantization makes inference", return tensors="pt" .to "cuda" def measure checkpoint path : torch.cuda.empty cache torch.cuda.reset peak memory stats model = AutoModelForCausalLM.from pretrained path, device map="cuda:0", dtype=torch.float16, .eval def generate current model : with torch.inference mode : return current model.generate inputs, max new tokens=32, do sample=False, pad token id=tokenizer.eos token id, generate model warm up torch.cuda.synchronize times = for in range 3 : start = time.perf counter ids = generate model torch.cuda.synchronize times.append time.perf counter - start result = { "median seconds": sorted times 1 , "peak allocated gib": torch.cuda.max memory allocated / 2 30, "output": tokenizer.decode ids 0 , skip special tokens=True , } del model gc.collect torch.cuda.empty cache return result for checkpoint in "facebook/opt-125m", "opt-125m-gptq" : print checkpoint, measure checkpoint checkpoint This is a small diagnostic benchmark. For deployment decisions, use the same prompts, output limits, warmup, and concurrency as the real workload. You should measure perplexity or task accuracy as well as sample generations; one matching greedy answer does not establish quality parity. Peak PyTorch allocation may omit memory held by other libraries, so check device-level memory too. A quantized model may require specialized kernels or a compatible inference engine. You may need specialized hardware for the best performance. You can compare the following measurements on the target platform to decide if quantization is worth the effort: - Checkpoint size and model-load memory - Peak memory during prefill and decode - Time to first token - Throughput at realistic batch sizes and concurrency Further Reading Below are some resources you may find useful: - PyTorch TorchAO documentation. Quantized Inference https://docs.pytorch.org/ao/stable/workflows/inference.html . This guide lists current weight-only and dynamic inference configurations. - PyTorch TorchAO documentation. quantize https://docs.pytorch.org/ao/stable/api reference/generated/torchao.quantization.quantize .html . This page documents the in-place post-training quantization API. - Elias Frantar et al. 2022 . GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers https://arxiv.org/abs/2210.17323 . This paper introduces the GPTQ algorithm. - Hugging Face Transformers documentation. GPTQ https://huggingface.co/docs/transformers/main/en/quantization/gptq . This guide covers quantization and checkpoint loading with GPT-QModel. - Ji Lin et al. 2023 . AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration https://arxiv.org/abs/2306.00978 . This paper introduces AWQ. - AWQ reference implementation https://github.com/mit-han-lab/llm-awq . By MIT Han Lab. This repository includes commands for searching AWQ parameters and producing real quantized weights. - AWQ conversion example https://github.com/mit-han-lab/llm-awq/blob/main/examples/convert to hf.py . In the reference repository. This shows how to package a raw quantized state dictionary with a Transformers configuration and tokenizer. - LLM Compressor https://github.com/vllm-project/llm-compressor . By the vLLM project. This toolkit provides GPTQ, AWQ, and other post-training compression recipes. - llama.cpp quantization https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md . In the llama.cpp repository. This guide documents conversion, quantization types, importance matrices, and execution. - Hugging Face Transformers documentation. GGUF https://huggingface.co/docs/transformers/main/en/quantization/gguf . This page explains the supported GGUF loading paths and fallback dequantization. - NVIDIA A100 Tensor Core GPU https://www.nvidia.com/en-us/data-center/a100/ . In the NVIDIA documentation. Its specifications compare peak throughput across FP16, INT8, and other numerical formats. - AMD Instinct MI300X Platform https://www.amd.com/en/products/accelerators/instinct/mi300/platform.html . In the AMD documentation. Its specifications provide another example of higher peak INT8 matrix throughput than FP16 throughput. Summary In this post, you learned how asymmetric affine quantization maps floating-point weights to integer codes using a scale and zero point. Per-tensor, per-channel, and per-group parameters trade metadata and kernel complexity for accuracy. Weight-only post-training quantization derives these parameters after training while keeping activations at higher precision. TorchAO provides PyTorch-native PTQ, while GPTQ and AWQ use calibration data to preserve model behavior at low bit widths. GPT-QModel, LLM Compressor, and the AWQ reference implementation provide workflows outside TorchAO. GGUF packages quantized tensors for runtimes such as llama.cpp . Quantization is useful only when the checkpoint format, numerical scheme, packed layout, and inference kernel work together. Evaluate quality, memory, latency, throughput, and compatibility on the actual deployment target.