# Weight-Only Quantization

> Source: <https://machinelearningmastery.com/weight-only-quantization/>
> Published: 2026-11-14 16:04:34+00:00

During autoregressive decoding, a transformer reads its weights for every new token. At small batch sizes, moving these weights can take more time than computation. Quantization reduces this cost by storing weights with fewer bits. For example, an INT8 weight uses half the raw storage of an FP16 or BF16 weight.

Quantization can also make computation faster. Many AI GPUs can perform low-precision matrix operations faster than FP16 operations. The actual speedup depends on the hardware, kernels, and model shapes.

Weight-only quantization stores selected weights as integers while keeping activations in FP16 or BF16. This post begins with the asymmetric affine mapping, then shows how to quantize and run models with PyTorch and other inference tools.

In this post, you will learn about:

- How weight-only quantization stores and reconstructs model weights
- How post-training methods and GGUF conversion produce deployable artifacts
- How to check the memory, speed, and quality effects of quantization

Let’s get started.

## Overview

This post is divided into five parts; they are:

- Asymmetric Affine Quantization
- Post-Training Quantization with PyTorch
- GPTQ and AWQ
- Quantizing a Model to GGUF
- Evaluating Weight-Only Quantization

**Note:** The code examples in this post use TorchAO and require a CUDA-enabled PyTorch installation. You can install the dependencies with:

`python -m pip install torchao transformers accelerate`

The GPTQ and AWQ sections have their own installation steps. GGUF conversion requires a built `llama.cpp` checkout.

## Asymmetric Affine Quantization

Integer quantization maps a range of real values onto a finite set of integer codes. An asymmetric affine quantizer uses a scale $s$ and a zero point $z$:

$$

q = \operatorname{clamp}\Big(\operatorname{round}\big(\frac{x}{s}\big) + z, q_{\min}, q_{\max}\Big)

$$

Dequantization reconstructs an approximate floating-point value:

$$

\hat{x} = s(q-z)

$$

For an unsigned $b$-bit range, $q_{\min}=0$ and $q_{\max}=2^b-1$. Given the minimum and maximum values to represent, one common choice is:

$$

\begin{aligned}

s &= \frac{x_{\max}-x_{\min}}{q_{\max}-q_{\min}} \\

z &= \operatorname{clamp}\Big(\operatorname{round}\big(q_{\min}-\frac{x_{\min}}{s}\big),q_{\min},q_{\max}\Big)

\end{aligned}

$$

The scale determines the distance between adjacent representable values. The zero point is the integer code for real zero. Including zero in the real range ensures that it can be represented exactly.

```
Quantize and reconstruct a tensor to show the error from integer rounding.

import torch

def quantize_asymmetric(x, bits=8):
    if not 1 <= bits <= 8:
        raise ValueError("this uint8 example supports 1 to 8 bits")
    if not x.is_floating_point():
        raise TypeError("x must be floating point")
    # max and min of quantized (unsigned int) values
    qmin = 0
    qmax = 2**bits - 1

    # max and min of actual (float) values
    xmin = torch.minimum(x.amin(), x.new_zeros(()))
    xmax = torch.maximum(x.amax(), x.new_zeros(()))

    # compute the scale and zero point, then quantize the actual values
    scale = (xmax - xmin) / (qmax - qmin)
    scale = scale.clamp_min(torch.finfo(x.dtype).eps)
    zero_point = torch.round(qmin - xmin / scale).clamp(qmin, qmax)

    q = torch.round(x / scale + zero_point).clamp(qmin, qmax)
    return q.to(torch.uint8), scale, zero_point.to(torch.int32)

def dequantize(q, scale, zero_point):
    return scale * (q.float() - zero_point.float())

x = torch.tensor([-2.3, -0.2, 0.0, 0.7, 3.1])
q, scale, zero_point = quantize_asymmetric(x)
x_hat = dequantize(q, scale, zero_point)

print("scale:", scale.item())
print("zero point:", zero_point.item())
print("quantized:", q)
print("maximum error:", (x - x_hat).abs().max().item())

12345678910111213141516171819202122232425262728293031323334

import torch def quantize_asymmetric(x, bits=8):    if not 1 <= bits <= 8:        raise ValueError("this uint8 example supports 1 to 8 bits")    if not x.is_floating_point():        raise TypeError("x must be floating point")    # max and min of quantized (unsigned int) values    qmin = 0    qmax = 2**bits - 1     # max and min of actual (float) values    xmin = torch.minimum(x.amin(), x.new_zeros(()))    xmax = torch.maximum(x.amax(), x.new_zeros(()))     # compute the scale and zero point, then quantize the actual values    scale = (xmax - xmin) / (qmax - qmin)    scale = scale.clamp_min(torch.finfo(x.dtype).eps)    zero_point = torch.round(qmin - xmin / scale).clamp(qmin, qmax)     q = torch.round(x / scale + zero_point).clamp(qmin, qmax)    return q.to(torch.uint8), scale, zero_point.to(torch.int32) def dequantize(q, scale, zero_point):    return scale * (q.float() - zero_point.float()) x = torch.tensor([-2.3, -0.2, 0.0, 0.7, 3.1])q, scale, zero_point = quantize_asymmetric(x)x_hat = dequantize(q, scale, zero_point) print("scale:", scale.item())print("zero point:", zero_point.item())print("quantized:", q)print("maximum error:", (x - x_hat).abs().max().item())
```

The example demonstrates the numerical mapping, not efficient storage. A `uint8` tensor always uses one byte per element, even if `bits` is less than eight. A deployment format must pack sub-byte values to obtain the expected storage reduction. This example uses one scale and zero point for the whole tensor. Per-channel and per-group quantization often preserve LLM weight quality better because each channel or group gets its own range.

For a linear layer $\mathbf{y}=\mathbf{x}\mathbf{W}^\top$, weight-only quantization stores an integer matrix $\mathbf{Q}$ plus scales and zero points instead of $\mathbf{W}$:

$$

\mathbf{y} \approx \mathbf{x}\big(s(\mathbf{Q}-z)\big)^\top

$$

The input remains floating point. An efficient kernel would use the quantized weights directly instead of reconstructing the floating-point copy of $\mathbf{W}$ like the equation shows.

The asymmetric affine mapping is the explanatory model used in this post. Production kernels and checkpoint formats may use a fixed zero point or another encoding. The runtime must support the exact quantization scheme and packed layout stored in the checkpoint.

## Post-Training Quantization with PyTorch

PyTorch’s current quantization development is centered in TorchAO. Its `quantize_()` function replaces the weights of matching `torch.nn.Linear` modules in place. Applying it to an evaluated, trained model is post-training quantization: no training step is performed.

The following example requests asymmetric, per-channel INT8 weight-only quantization. The activations remain BF16:

``` python
Apply per-channel INT8 weight-only quantization with TorchAO.

import torch
import torch.nn as nn
from torchao.quantization import (
    IntxWeightOnlyConfig,
    MappingType,
    PerAxis,
    quantize_,
)

if not torch.cuda.is_available():
    raise RuntimeError("This example requires a CUDA GPU")

model = nn.Sequential(
    nn.Linear(2048, 2048, bias=False),
    nn.ReLU(),
    nn.Linear(2048, 2048, bias=False),
).eval().to(device="cuda", dtype=torch.bfloat16)

config = IntxWeightOnlyConfig(
    weight_dtype=torch.int8,
    granularity=PerAxis(0),
    mapping_type=MappingType.ASYMMETRIC,
)
quantize_(model, config)

x = torch.randn(1, 2048, device="cuda", dtype=torch.bfloat16)
with torch.inference_mode():
    output = model(x)

print(output.shape)

123456789101112131415161718192021222324252627282930

import torchimport torch.nn as nnfrom torchao.quantization import (    IntxWeightOnlyConfig,    MappingType,    PerAxis,    quantize_,) if not torch.cuda.is_available():    raise RuntimeError("This example requires a CUDA GPU") model = nn.Sequential(    nn.Linear(2048, 2048, bias=False),    nn.ReLU(),    nn.Linear(2048, 2048, bias=False),).eval().to(device="cuda", dtype=torch.bfloat16) config = IntxWeightOnlyConfig(    weight_dtype=torch.int8,    granularity=PerAxis(0),    mapping_type=MappingType.ASYMMETRIC,)quantize_(model, config) x = torch.randn(1, 2048, device="cuda", dtype=torch.bfloat16)with torch.inference_mode():    output = model(x) print(output.shape)
```

`PerAxis(0)` assigns separate quantization parameters to the rows of each linear weight matrix. `IntxWeightOnlyConfig` also supports per-group granularity. Its available packing formats and optimized kernels depend on the TorchAO version and target device.

TorchAO also provides hardware-oriented presets such as `Int8WeightOnlyConfig` and `Int4WeightOnlyConfig`. These presets may use a different mapping from the explicit asymmetric example. You should use the configuration documented for the target hardware, then benchmark the complete model. A mathematically valid configuration is not necessarily the fastest one.

The `transformers` library can apply a TorchAO configuration while loading a model. For example:

``` python
Load a causal model with a TorchAO quantization configuration.

import torch
from torchao.quantization import Int4WeightOnlyConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, TorchAoConfig

model_id = "Qwen/Qwen2.5-0.5B-Instruct"
quantization_config = TorchAoConfig(
    quant_type=Int4WeightOnlyConfig(
        group_size=128,
        int4_packing_format="tile_packed_to_4d",
        int4_choose_qparams_algorithm="hqq",
    )
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
    quantization_config=quantization_config,
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

inputs = tokenizer("Explain post-training quantization.", return_tensors="pt")
inputs = inputs.to(model.device)
with torch.inference_mode():
    generated = model.generate(**inputs, max_new_tokens=32)

print(tokenizer.decode(generated[0], skip_special_tokens=True))

123456789101112131415161718192021222324252627

import torchfrom torchao.quantization import Int4WeightOnlyConfigfrom transformers import AutoModelForCausalLM, AutoTokenizer, TorchAoConfig model_id = "Qwen/Qwen2.5-0.5B-Instruct"quantization_config = TorchAoConfig(    quant_type=Int4WeightOnlyConfig(        group_size=128,        int4_packing_format="tile_packed_to_4d",        int4_choose_qparams_algorithm="hqq",    )) model = AutoModelForCausalLM.from_pretrained(    model_id,    dtype=torch.bfloat16,    device_map="auto",    quantization_config=quantization_config,)tokenizer = AutoTokenizer.from_pretrained(model_id) inputs = tokenizer("Explain post-training quantization.", return_tensors="pt")inputs = inputs.to(model.device)with torch.inference_mode():    generated = model.generate(**inputs, max_new_tokens=32) print(tokenizer.decode(generated[0], skip_special_tokens=True))
```

This is calibration-free weight-only PTQ. The explicit HQQ parameter choice and tiled packing avoid an optional MSLK kernel dependency in TorchAO 0.17 on the tested RTX 4090. Other TorchAO versions and devices may support different layouts. Static activation quantization is a different workflow: it prepares an exported model, runs representative inputs to collect activation ranges, and then converts the model.

## GPTQ and AWQ

Round-to-nearest quantization treats each weight locally. GPTQ and AWQ use calibration activations to reduce the effect of quantization error on model outputs.

Rounding each weight independently ignores how often and how strongly its input channel contributes to outputs. Calibration examples let a quantizer estimate this effect and reduce output error, though the result depends on how representative those examples are.

GPTQ quantizes weights layer by layer using approximate second-order information. As weights are quantized, it adjusts the remaining weights to compensate for earlier error. The `transformers` library now uses GPT-QModel as its GPTQ backend. Install the required packages to use GPTQ for quantization:

```
Install the GPTQ conversion dependencies.

python -m pip install setuptools --upgrade wheel accelerate optimum transformers
python -m pip install gptqmodel --no-build-isolation

12

python -m pip install setuptools --upgrade wheel accelerate optimum transformerspython -m pip install gptqmodel --no-build-isolation
```

Then you can quantize and save a small model. The `GPTQConfig` setting `dataset="c4"` provides input samples for calibration. Setting `sym=False` requests the asymmetric form. Below is an example of quantizing a small model to INT4:

``` python
Calibrate and save an INT4 GPTQ checkpoint.

from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig

model_id = "facebook/opt-125m"
tokenizer = AutoTokenizer.from_pretrained(model_id)
quantization_config = GPTQConfig(
    bits=4,  # 2, 3, 4, or 8
    group_size=128,
    dataset="c4",
    tokenizer=tokenizer,
    sym=False,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    quantization_config=quantization_config,
)
model.save_pretrained("opt-125m-gptq")
tokenizer.save_pretrained("opt-125m-gptq")
print("Saved the GPTQ checkpoint to opt-125m-gptq")

1234567891011121314151617181920

from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig model_id = "facebook/opt-125m"tokenizer = AutoTokenizer.from_pretrained(model_id)quantization_config = GPTQConfig(    bits=4,  # 2, 3, 4, or 8    group_size=128,    dataset="c4",    tokenizer=tokenizer,    sym=False,) model = AutoModelForCausalLM.from_pretrained(    model_id,    device_map="auto",    quantization_config=quantization_config,)model.save_pretrained("opt-125m-gptq")tokenizer.save_pretrained("opt-125m-gptq")print("Saved the GPTQ checkpoint to opt-125m-gptq")
```

To load the saved GPTQ checkpoint in a fresh process and run one inference:

``` python
Reload the saved GPTQ checkpoint and generate text.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "opt-125m-gptq"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(
    checkpoint,
    device_map="auto",
)
model.eval()

inputs = tokenizer("Weight quantization makes inference", return_tensors="pt")
inputs = inputs.to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=32,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

12345678910111213141516171819202122

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer checkpoint = "opt-125m-gptq"tokenizer = AutoTokenizer.from_pretrained(checkpoint)model = AutoModelForCausalLM.from_pretrained(    checkpoint,    device_map="auto",)model.eval() inputs = tokenizer("Weight quantization makes inference", return_tensors="pt")inputs = inputs.to(model.device) with torch.inference_mode():    output = model.generate(        **inputs,        max_new_tokens=32,        pad_token_id=tokenizer.eos_token_id,    ) print(tokenizer.decode(output[0], skip_special_tokens=True))
```

Asymmetric GPTQ checkpoints have fewer compatible kernels than the default mapping. For example, the Marlin backend does not support asymmetric GPTQ. You should choose the quantization scheme and inference backend together.

Activation-aware Weight Quantization (AWQ) is another quantization method. It identifies weights that are important under representative activations and searches for scaling factors that protect them before low-bit quantization. The `--model_path` argument points to the original, unquantized model in Hugging Face format. From a checkout of the reference `llm-awq` implementation with its dependencies installed, run the search first and then create the real INT4 checkpoint:

Install the reference implementation in a separate environment before running its commands:

```
Install AutoAWQ and the Hugging Face Hub client.

git clone https://github.com/mit-han-lab/llm-awq.git
cd llm-awq
python -m pip install -e .
python -m pip install huggingface_hub

1234

git clone https://github.com/mit-han-lab/llm-awq.gitcd llm-awqpython -m pip install -e .python -m pip install huggingface_hub
```

The reference project documents Python 3.10 for this environment. Use its [installation instructions](https://github.com/mit-han-lab/llm-awq#install) for the matching CUDA extensions. Download the model with

```
hf download TinyLlama/TinyLlama-1.1B-Chat-v1.0 --local-dir models/tinyllama

1

hf download TinyLlama/TinyLlama-1.1B-Chat-v1.0 --local-dir models/tinyllama
```

then run the following commands:

```
Search for AWQ scales and quantize the TinyLlama checkpoint.

# Search for AWQ scaling parameters.
python -m awq.entry \
  --model_path models/tinyllama \
  --w_bit 4 \
  --q_group_size 128 \
  --run_awq \
  --dump_awq awq_cache/tinyllama-w4-g128.pt

# Create the quantized checkpoint.
python -m awq.entry \
  --model_path models/tinyllama \
  --w_bit 4 \
  --q_group_size 128 \
  --load_awq awq_cache/tinyllama-w4-g128.pt \
  --q_backend real \
  --dump_quant quant_cache/tinyllama-w4-g128-awq-v2.pt

12345678910111213141516

# Search for AWQ scaling parameters.python -m awq.entry \  --model_path models/tinyllama \  --w_bit 4 \  --q_group_size 128 \  --run_awq \  --dump_awq awq_cache/tinyllama-w4-g128.pt # Create the quantized checkpoint.python -m awq.entry \  --model_path models/tinyllama \  --w_bit 4 \  --q_group_size 128 \  --load_awq awq_cache/tinyllama-w4-g128.pt \  --q_backend real \  --dump_quant quant_cache/tinyllama-w4-g128-awq-v2.pt
```

The `--run_awq` command obtains its calibration input internally. It uses 128 samples from the validation split of `mit-han-lab/pile-val-backup` and forms sequences of 512 tokens. This reference CLI does not expose an argument for selecting another text dataset. Omitting `--no_zero_point` keeps the implementation’s zero-point quantization.

The raw `.pt` file produced by `--dump_quant` is not a complete Hugging Face checkpoint. It contains the quantized weights but not the model configuration or tokenizer. The following script packages them together in a local directory:

```
Package an AWQ weight file with its model configuration and tokenizer.

from pathlib import Path
import shutil

from transformers import AutoConfig, AutoTokenizer, AwqConfig

source_model = "models/tinyllama"
quantized_weights = "quant_cache/tinyllama-w4-g128-awq-v2.pt"
output_dir = Path("models/tinyllama-awq")
output_dir.mkdir(parents=True, exist_ok=True)

awq_config = AwqConfig(
    bits=4,
    group_size=128,
    zero_point=True,
    backend="llm-awq",
    version="gemv",
)

config = AutoConfig.from_pretrained(source_model)
config.quantization_config = awq_config
config.save_pretrained(output_dir)

tokenizer = AutoTokenizer.from_pretrained(source_model)
tokenizer.save_pretrained(output_dir)

shutil.copyfile(
    quantized_weights,
    output_dir / "pytorch_model.bin",
)

1234567891011121314151617181920212223242526272829

from pathlib import Pathimport shutil from transformers import AutoConfig, AutoTokenizer, AwqConfig source_model = "models/tinyllama"quantized_weights = "quant_cache/tinyllama-w4-g128-awq-v2.pt"output_dir = Path("models/tinyllama-awq")output_dir.mkdir(parents=True, exist_ok=True) awq_config = AwqConfig(    bits=4,    group_size=128,    zero_point=True,    backend="llm-awq",    version="gemv",) config = AutoConfig.from_pretrained(source_model)config.quantization_config = awq_configconfig.save_pretrained(output_dir) tokenizer = AutoTokenizer.from_pretrained(source_model)tokenizer.save_pretrained(output_dir) shutil.copyfile(    quantized_weights,    output_dir / "pytorch_model.bin",)
```

The `transformers` library can then identify the AWQ format from the checkpoint’s `quantization_config`, load the quantized weights, and run inference:

``` python
Load the packaged AWQ checkpoint and generate text.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "models/tinyllama-awq"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
)
model.eval()

inputs = tokenizer("Explain weight quantization.", return_tensors="pt")
inputs = inputs.to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

12345678910111213141516171819202122

import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer model_id = "models/tinyllama-awq"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(    model_id,    device_map="auto",)model.eval() inputs = tokenizer("Explain weight quantization.", return_tensors="pt")inputs = inputs.to(model.device) with torch.inference_mode():    output = model.generate(        **inputs,        max_new_tokens=64,        pad_token_id=tokenizer.eos_token_id,    ) print(tokenizer.decode(output[0], skip_special_tokens=True))
```

The model architecture, group size, zero-point convention, packed layout, and installed AWQ backend must be compatible.

Quantizing a large model can take substantial time and memory. Before starting, check whether a compatible quantized checkpoint already exists. If you quantize it yourself, save the tokenizer and complete quantization configuration with the packed weights.

## Quantizing a Model to GGUF

GGUF is a container format used by the `llama.cpp` ecosystem. It stores model metadata, tokenizer information, tensors, and their types in one or more files. GGUF supports many quantization types; it is not itself one quantization algorithm.

The `llama.cpp` workflow has two stages. First, convert the original Hugging Face model to a high-precision GGUF file (such as bfloat16), then quantize that file. The following example uses the original, unquantized `TinyLlama/TinyLlama-1.1B-Chat-v1.0` checkpoint. From a built `llama.cpp` checkout:

Install the conversion script’s Python dependencies in a separate environment and build the binaries first. The conversion requirements may select a different PyTorch build from the one used for the CUDA examples earlier in this post. This CUDA build needs CMake, a C++ compiler, and the CUDA toolkit; omit `-DGGML_CUDA=ON` for a CPU build.

```
Clone and build llama.cpp with GGUF conversion support.

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
python -m pip install -r requirements/requirements-convert_hf_to_gguf.txt
python -m pip install huggingface_hub
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release
hf download TinyLlama/TinyLlama-1.1B-Chat-v1.0 --local-dir models/tinyllama

1234567

git clone https://github.com/ggml-org/llama.cpp.gitcd llama.cpppython -m pip install -r requirements/requirements-convert_hf_to_gguf.txtpython -m pip install huggingface_hubcmake -B build -DGGML_CUDA=ONcmake --build build --config Releasehf download TinyLlama/TinyLlama-1.1B-Chat-v1.0 --local-dir models/tinyllama
Convert TinyLlama to GGUF and quantize it to Q4_K_M.

python3 convert_hf_to_gguf.py models/tinyllama \
  --outfile tinyllama-bf16.gguf \
  --outtype bf16

./build/bin/llama-quantize tinyllama-bf16.gguf tinyllama-Q4_K_M.gguf Q4_K_M

12345

python3 convert_hf_to_gguf.py models/tinyllama \  --outfile tinyllama-bf16.gguf \  --outtype bf16 ./build/bin/llama-quantize tinyllama-bf16.gguf tinyllama-Q4_K_M.gguf Q4_K_M
```

`Q4_K_M` is a mixed GGUF quantization type that uses different encodings for different tensors. Its file size includes scales, metadata, and tensors stored at other precisions, so it is not exactly four bits per model parameter. Its name does not mean every value follows the affine formula shown earlier.

You can use `llama-cli` to load the quantized GGUF file and generate text directly from its packed weights:

```
Run the quantized GGUF model with llama.cpp.

./build/bin/llama-cli \
  -m tinyllama-Q4_K_M.gguf \
  -p "Explain weight-only quantization in one paragraph." \
  -n 128 --single-turn

1234

./build/bin/llama-cli \  -m tinyllama-Q4_K_M.gguf \  -p "Explain weight-only quantization in one paragraph." \  -n 128 --single-turn
```

This confirms that `llama.cpp` can generate from the packed file. TinyLlama may give an inaccurate explanation, so use separate quality checks for your task.

Some GGUF models are available on the Hugging Face Hub and can be run directly:

```
Run a GGUF model directly from the Hugging Face Hub.

./build/bin/llama-cli -hf Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M \
  -p "What is a quantized model?" -n 64 --single-turn

12

./build/bin/llama-cli -hf Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M \  -p "What is a quantized model?" -n 64 --single-turn
```

If you prefer to use Python code, `transformers` also accepts a GGUF filename for supported architectures:

```
Install the Hugging Face library with GGUF loading support.

python -m pip install --upgrade transformers gguf

1

python -m pip install --upgrade transformers gguf
```

Use a recent `transformers` release for the Qwen3.5 example below. Loading may dequantize tensors and require much more memory than the GGUF file size suggests.

```
Load a GGUF checkpoint through the Hugging Face library.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "unsloth/Qwen3.5-4B-GGUF"
filename = "Qwen3.5-4B-Q4_K_M.gguf"

model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)
tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
print(type(model).__name__, type(tokenizer).__name__)

12345678

from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "unsloth/Qwen3.5-4B-GGUF"filename = "Qwen3.5-4B-Q4_K_M.gguf" model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)print(type(model).__name__, type(tokenizer).__name__)
```

But you should not assume that `transformers` preserves compressed inference. It keeps GGUF blocks packed only for a narrow set of model architectures, devices, and quantization types; other combinations are dequantized while loading. You can use `llama.cpp` when broad GGUF compatibility and direct execution of packed weights are required. You should quantize from the highest-quality source available. Requantizing an already quantized model compounds approximation error, and `llama-quantize` warns that it can severely reduce quality.

## Evaluating Weight-Only Quantization

Evaluate the quantized model on representative inputs. If quality degrades beyond your application’s tolerance, try a different calibration set, quantizer, or bit width.

Some quantizers use calibration examples to reduce output error between the original and quantized model. That proxy does not replace application quality metrics. Useful checks include:

- Perplexity on representative text
- Accuracy or task-specific scores
- Compare greedy generations side by side
- Compare sampled generations across several seeds, prompt lengths, and temperatures

After creating `opt-125m-gptq` with the earlier command, run this script on one CUDA GPU. It loads and measures the FP16 baseline and the GPTQ checkpoint *one at a time* so their peak allocated memory measurements are comparable. The timing includes `generate()` but excludes model loading and tokenization:

``` python
Compare memory use and generation time across model checkpoints.

import gc
import time

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

if not torch.cuda.is_available():
    raise RuntimeError("This comparison needs a CUDA GPU and a GPTQ CUDA kernel")

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m")
inputs = tokenizer("Weight quantization makes inference", return_tensors="pt").to("cuda")

def measure_checkpoint(path):
    torch.cuda.empty_cache()
    torch.cuda.reset_peak_memory_stats()
    model = AutoModelForCausalLM.from_pretrained(
        path, device_map="cuda:0", dtype=torch.float16,
    ).eval()

    def generate(current_model):
        with torch.inference_mode():
            return current_model.generate(
                **inputs, max_new_tokens=32, do_sample=False,
                pad_token_id=tokenizer.eos_token_id,
            )

    generate(model)  # warm up
    torch.cuda.synchronize()
    times = []
    for _ in range(3):
        start = time.perf_counter()
        ids = generate(model)
        torch.cuda.synchronize()
        times.append(time.perf_counter() - start)

    result = {
        "median_seconds": sorted(times)[1],
        "peak_allocated_gib": torch.cuda.max_memory_allocated() / 2**30,
        "output": tokenizer.decode(ids[0], skip_special_tokens=True),
    }
    del model
    gc.collect()
    torch.cuda.empty_cache()
    return result

for checkpoint in ("facebook/opt-125m", "opt-125m-gptq"):
    print(checkpoint, measure_checkpoint(checkpoint))

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950

import gcimport time import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer  if not torch.cuda.is_available():    raise RuntimeError("This comparison needs a CUDA GPU and a GPTQ CUDA kernel") tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m")inputs = tokenizer("Weight quantization makes inference", return_tensors="pt").to("cuda")  def measure_checkpoint(path):    torch.cuda.empty_cache()    torch.cuda.reset_peak_memory_stats()    model = AutoModelForCausalLM.from_pretrained(        path, device_map="cuda:0", dtype=torch.float16,    ).eval()     def generate(current_model):        with torch.inference_mode():            return current_model.generate(                **inputs, max_new_tokens=32, do_sample=False,                pad_token_id=tokenizer.eos_token_id,            )     generate(model)  # warm up    torch.cuda.synchronize()    times = []    for _ in range(3):        start = time.perf_counter()        ids = generate(model)        torch.cuda.synchronize()        times.append(time.perf_counter() - start)     result = {        "median_seconds": sorted(times)[1],        "peak_allocated_gib": torch.cuda.max_memory_allocated() / 2**30,        "output": tokenizer.decode(ids[0], skip_special_tokens=True),    }    del model    gc.collect()    torch.cuda.empty_cache()    return result  for checkpoint in ("facebook/opt-125m", "opt-125m-gptq"):    print(checkpoint, measure_checkpoint(checkpoint))
```

This is a small diagnostic benchmark. For deployment decisions, use the same prompts, output limits, warmup, and concurrency as the real workload. You should measure perplexity or task accuracy as well as sample generations; one matching greedy answer does not establish quality parity. Peak PyTorch allocation may omit memory held by other libraries, so check device-level memory too.

A quantized model may require specialized kernels or a compatible inference engine. You may need specialized hardware for the best performance. You can compare the following measurements on the target platform to decide if quantization is worth the effort:

- Checkpoint size and model-load memory
- Peak memory during prefill and decode
- Time to first token
- Throughput at realistic batch sizes and concurrency

## Further Reading

Below are some resources you may find useful:

- PyTorch TorchAO documentation. [Quantized Inference](https://docs.pytorch.org/ao/stable/workflows/inference.html) . This guide lists current weight-only and dynamic inference configurations.
- PyTorch TorchAO documentation. [quantize_](https://docs.pytorch.org/ao/stable/api_reference/generated/torchao.quantization.quantize_.html) . This page documents the in-place post-training quantization API.
- Elias Frantar et al. (2022). [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://arxiv.org/abs/2210.17323) . This paper introduces the GPTQ algorithm.
- Hugging Face Transformers documentation. [GPTQ](https://huggingface.co/docs/transformers/main/en/quantization/gptq) . This guide covers quantization and checkpoint loading with GPT-QModel.
- Ji Lin et al. (2023). [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://arxiv.org/abs/2306.00978) . This paper introduces AWQ.
- [AWQ reference implementation](https://github.com/mit-han-lab/llm-awq) . By MIT Han Lab. This repository includes commands for searching AWQ parameters and producing real quantized weights.
- [AWQ conversion example](https://github.com/mit-han-lab/llm-awq/blob/main/examples/convert_to_hf.py) . In the reference repository. This shows how to package a raw quantized state dictionary with a Transformers configuration and tokenizer.
- [LLM Compressor](https://github.com/vllm-project/llm-compressor) . By the vLLM project. This toolkit provides GPTQ, AWQ, and other post-training compression recipes.
- [llama.cpp quantization](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) . In the llama.cpp repository. This guide documents conversion, quantization types, importance matrices, and execution.
- Hugging Face Transformers documentation. [GGUF](https://huggingface.co/docs/transformers/main/en/quantization/gguf) . This page explains the supported GGUF loading paths and fallback dequantization.
- [NVIDIA A100 Tensor Core GPU](https://www.nvidia.com/en-us/data-center/a100/) . In the NVIDIA documentation. Its specifications compare peak throughput across FP16, INT8, and other numerical formats.
- [AMD Instinct MI300X Platform](https://www.amd.com/en/products/accelerators/instinct/mi300/platform.html) . In the AMD documentation. Its specifications provide another example of higher peak INT8 matrix throughput than FP16 throughput.

## Summary

In this post, you learned how asymmetric affine quantization maps floating-point weights to integer codes using a scale and zero point. Per-tensor, per-channel, and per-group parameters trade metadata and kernel complexity for accuracy. Weight-only post-training quantization derives these parameters after training while keeping activations at higher precision.

TorchAO provides PyTorch-native PTQ, while GPTQ and AWQ use calibration data to preserve model behavior at low bit widths. GPT-QModel, LLM Compressor, and the AWQ reference implementation provide workflows outside TorchAO. GGUF packages quantized tensors for runtimes such as `llama.cpp`.

Quantization is useful only when the checkpoint format, numerical scheme, packed layout, and inference kernel work together. Evaluate quality, memory, latency, throughput, and compatibility on the actual deployment target.