Weight-Only Quantization focused on weights, which are fixed after training and can be quantized once before deployment. This post turns to tensors created during inference. Activations depend on the input token, sequence position, and previous layers. Their ranges can change from request to request, and a few channels may contain persistent outliers. This makes activation quantization harder than weight-only quantization.
In this post, you will learn about:
- Why dynamic activations are harder to quantize than fixed weights
- How W8A8 methods handle activation ranges
- How floating-point quantization differs from integer quantization
Let’s get started.
Overview #
This post is divided into three parts; they are:
- Why Activations Are Hard to Quantize
- W8A8 Inference and SmoothQuant
- Floating-Point Quantization
Why Activations Are Hard to Quantize #
The post on weight-only quantization introduced scales, zero points, and quantization granularity for fixed weights. The same concepts apply to activations, but the quantization parameters may now need to follow values that do not exist until inference begins.
Consider an activation tensor entering a linear layer. A static activation quantizer uses scales chosen during calibration. It is fast because the scales are already known, but a deployment input outside the calibration range may be clipped. A dynamic activation quantizer measures the tensor at runtime. It adapts to each input but adds reduction and scaling work to the inference path.
An activation quantizer can be written as follows:
Quantize an activation tensor with scales computed at runtime.
import torch
def quantize_activation_per_token(x):
qmax = 127
scale = x.abs().amax(dim=-1, keepdim=True) / qmax
scale = scale.clamp_min(torch.finfo(x.dtype).eps)
q = torch.round(x / scale).clamp(-qmax, qmax)
return q.to(torch.int8), scale
x = torch.randn(2, 32, 1024)
q, scale = quantize_activation_per_token(x)
x_hat = q.float() * scale
print("mean absolute error:", (x - x_hat).abs().mean().item())
12345678910111213
import torch def quantize_activation_per_token(x): qmax = 127 scale = x.abs().amax(dim=-1, keepdim=True) / qmax scale = scale.clamp_min(torch.finfo(x.dtype).eps) q = torch.round(x / scale).clamp(-qmax, qmax) return q.to(torch.int8), scale x = torch.randn(2, 32, 1024)q, scale = quantize_activation_per_token(x)x_hat = q.float() * scaleprint("mean absolute error:", (x - x_hat).abs().mean().item())
For an activation shaped [batch, sequence, hidden], each vector along the final dimension represents one token. Reducing over that dimension produces a scale shaped [batch, sequence, 1]: one scale shared by all hidden features of each token. A linear layer treats these token vectors as matrix rows, so this scheme is also called per-row quantization.
Dynamic per-token scaling is a common implementation for transformer activations. It follows token-to-token range changes more closely than one scale for the entire tensor. The example uses signed symmetric INT8, so the zero point is fixed at zero instead of being calculated for every token. This simplifies the quantization and matrix-multiplication arithmetic. The trade-off is that the absolute-maximum reduction and scale calculation must run whenever a new activation tensor is produced.
Transformer activations often have outlier values in the vector. Covering the entire range would make the step size coarse for ordinary values. Clipping improves resolution for ordinary values but destroys the outlier. The best choice depends on how the following weights use that vector.
Activation distributions also differ by layer and token. Static activation quantization uses calibration data to predict the ranges of future runtime tensors, then select the most appropriate scale and zero point. Dynamic activation quantization avoids fixed calibration ranges but pays the runtime measurement cost.
W8A8 Inference and SmoothQuant #
Here, W8A8 means INT8 weights and INT8 activations for matrix multiplication. When the hardware provides efficient integer matrix multiplication, this can reduce memory traffic and increase arithmetic throughput. Biases, normalization, residuals, or accumulators may remain at higher precision. W8A8 by itself names bit widths; another workflow could use an FP8 format.
SmoothQuant addresses activation outliers by moving some quantization difficulty from activations into weights. For a linear layer $\mathbf{y}=\mathbf{x}\mathbf{W}^\top$, choose a positive scale vector $\mathbf{s}$ over input channels:
$$
\mathbf{y} = (\mathbf{x} \operatorname{diag}(\mathbf{s})^{-1})
(\mathbf{W} \operatorname{diag}(\mathbf{s}))^\top
$$
The transformation is mathematically equivalent before quantization. Large activation channels are divided by larger scales, making the activation range smoother. The corresponding weight columns are multiplied by the same scales. Weights are static and usually tolerate this uneven range better with per-channel quantization.
This differs from AWQ in the previous post. AWQ uses calibration activations to protect important weights in a weight-only format such as W4A16; the runtime activations remain in a higher-precision format. SmoothQuant prepares both sides for W8A8 execution: it changes the stored weights and makes the runtime activations easier to quantize to INT8.
The following code checks the equivalence:
Check the algebraic equivalence used by SmoothQuant.
import torch
import torch.nn.functional as F
x = torch.randn(4, 128)
weight = torch.randn(256, 128)
smooth_scale = torch.rand(128) + 0.5
reference = F.linear(x, weight)
smoothed_x = x / smooth_scale
smoothed_weight = weight * smooth_scale
transformed = F.linear(smoothed_x, smoothed_weight)
torch.testing.assert_close(transformed, reference, rtol=1e-5, atol=1e-5)
12345678910111213
import torchimport torch.nn.functional as F x = torch.randn(4, 128)weight = torch.randn(256, 128)smooth_scale = torch.rand(128) + 0.5 reference = F.linear(x, weight)smoothed_x = x / smooth_scalesmoothed_weight = weight * smooth_scaletransformed = F.linear(smoothed_x, smoothed_weight) torch.testing.assert_close(transformed, reference, rtol=1e-5, atol=1e-5)
This only demonstrates the exact rescaling. A SmoothQuant implementation derives smooth_scale from weight magnitudes and calibration activations, folds scales into adjacent operations where possible, and quantizes the transformed tensors.
One practical way to create a reusable INT8 weight-and-activation checkpoint is LLM Compressor, a library in the vLLM project. The following workflow uses transformers to load the model, LLM Compressor to apply SmoothQuant and INT8 W8A8 quantization, and then saves the result for vLLM inference:
Calibrate and save an INT8 W8A8 checkpoint with LLM Compressor.
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier
from llmcompressor.modifiers.transform.smoothquant import SmoothQuantModifier
model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
output_dir = "models/tinyllama-w8a8"
num_calibration_samples = 128
max_sequence_length = 512
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
dtype="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
dataset = load_dataset(
"HuggingFaceH4/ultrachat_200k",
split="train_sft",
)
dataset = dataset.shuffle(seed=42).select(range(num_calibration_samples))
def preprocess(example):
text = tokenizer.apply_chat_template(example["messages"], tokenize=False)
return {"text": text}
def tokenize(example):
return tokenizer(
example["text"],
max_length=max_sequence_length,
truncation=True,
add_special_tokens=False,
)
dataset = dataset.map(preprocess)
dataset = dataset.map(tokenize, remove_columns=dataset.column_names)
recipe = [
SmoothQuantModifier(smoothing_strength=0.8),
GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head"]),
]
oneshot(
model=model,
dataset=dataset,
recipe=recipe,
max_seq_length=max_sequence_length,
num_calibration_samples=num_calibration_samples,
)
model.save_pretrained(output_dir, save_compressed=True)
tokenizer.save_pretrained(output_dir)
print(f"Saved the INT8 checkpoint to {output_dir}")
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859
from datasets import load_datasetfrom transformers import AutoModelForCausalLM, AutoTokenizer from llmcompressor import oneshotfrom llmcompressor.modifiers.gptq import GPTQModifierfrom llmcompressor.modifiers.transform.smoothquant import SmoothQuantModifier model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"output_dir = "models/tinyllama-w8a8"num_calibration_samples = 128max_sequence_length = 512 model = AutoModelForCausalLM.from_pretrained( model_id, device_map="auto", dtype="auto",)tokenizer = AutoTokenizer.from_pretrained(model_id) dataset = load_dataset( "HuggingFaceH4/ultrachat_200k", split="train_sft",)dataset = dataset.shuffle(seed=42).select(range(num_calibration_samples)) def preprocess(example): text = tokenizer.apply_chat_template(example["messages"], tokenize=False) return {"text": text} def tokenize(example): return tokenizer( example["text"], max_length=max_sequence_length, truncation=True, add_special_tokens=False, ) dataset = dataset.map(preprocess)dataset = dataset.map(tokenize, remove_columns=dataset.column_names) recipe = [ SmoothQuantModifier(smoothing_strength=0.8), GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head"]),] oneshot( model=model, dataset=dataset, recipe=recipe, max_seq_length=max_sequence_length, num_calibration_samples=num_calibration_samples,) model.save_pretrained(output_dir, save_compressed=True)tokenizer.save_pretrained(output_dir)print(f"Saved the INT8 checkpoint to {output_dir}")
The saved checkpoint contains per-channel INT8 weights and instructions for dynamically quantizing activations during inference. The activations themselves are not stored in the checkpoint because they are produced and quantized per token at runtime.
vLLM can load the local checkpoint and use its W8A8 kernels without a separate quantization argument:
Load the saved INT8 W8A8 checkpoint and generate text with vLLM.
from vllm import LLM, SamplingParams
model = LLM(model="models/tinyllama-w8a8")
sampling_params = SamplingParams(temperature=0, min_tokens=16, max_tokens=64)
outputs = model.generate(
["Explain activation quantization in one paragraph."],
sampling_params,
)
print(outputs[0].outputs[0].text)
12345678910
from vllm import LLM, SamplingParams model = LLM(model="models/tinyllama-w8a8")sampling_params = SamplingParams(temperature=0, min_tokens=16, max_tokens=64) outputs = model.generate( ["Explain activation quantization in one paragraph."], sampling_params,)print(outputs[0].outputs[0].text)
This short generation checks that the checkpoint loads and runs. The small model’s answer may be inaccurate; judge quantization quality against the original model on representative evaluation prompts.
The checkpoint format, model architecture, vLLM version, and GPU must support the same W8A8 scheme. Current vLLM INT8 kernels target NVIDIA GPUs from Turing through Hopper; Blackwell users should choose a supported FP8 path instead.
W8A8 is not guaranteed to be faster than W4A16 or BF16. W8A8 can use integer matrix hardware efficiently, especially for prefill and larger batches. W4A16 reads fewer weight bytes and can be attractive for small-batch decode. You should benchmark both phases.
Floating-Point Quantization #
FP8 quantization does not simply crop bits from an FP32 value. In a common scaled FP8 scheme, the original tensor is divided by a scale and each result is rounded to a representable FP8 value:
$$
\begin{aligned}
q &= \operatorname{cast}_{\text{FP8}}(x/s), \
\hat{x} &= s\operatorname{cast}_{\text{FP16 or FP32}}(q)
\end{aligned}
$$
The scale must accompany the FP8 tensor or block so the runtime can interpret its values. FP8 is therefore still a quantized representation rather than an ordinary unscaled cast to a smaller storage type.
Unlike affine INT8, FP8 does not place representable values at one regular step size across the whole quantization group. Its exponent gives it nonuniform spacing: values near zero have finer steps, while larger values have wider steps. This covers a broader range of magnitudes, although rounding and overflow are still possible. Scaling places the source values in the most useful part of that FP8 range; it normally does not require an INT8-style zero-point shift.
The word “FP8” does not specify which FP8 layout is used. For example, E4M3 has four exponent bits and three significand bits, while E5M2 has five exponent bits and two significand bits. E4M3 provides more precision, whereas E5M2 provides a wider range. Which one can be used depends on the hardware and inference engine. FP8 also does not specify whether scaling is per tensor or per block, or how scales are calculated and stored.
FP4 is similar, but requires more aggressive scaling and appropriate hardware support.
The vLLM project’s LLM Compressor can convert a Hugging Face checkpoint to an FP8 checkpoint. Its FP8_DYNAMIC scheme uses round-to-nearest quantization for the fixed weights and derives their scales directly from the weight tensors. Activations are not available during conversion, so vLLM measures them and calculates per-token scales during inference. Neither operation needs representative input samples, which is why this basic workflow does not provide calibration data:
Save a dynamically quantized FP8 checkpoint.
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
output_dir = "models/tinyllama-w8a8-fp8"
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)
recipe = QuantizationModifier(
targets="Linear",
scheme="FP8_DYNAMIC",
ignore=["lm_head"],
)
oneshot(model=model, recipe=recipe)
model.save_pretrained(output_dir, save_compressed=True)
tokenizer.save_pretrained(output_dir)
print(f"Saved the FP8 checkpoint to {output_dir}")
123456789101112131415161718192021
from transformers import AutoModelForCausalLM, AutoTokenizer from llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifier model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"output_dir = "models/tinyllama-w8a8-fp8" model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto")tokenizer = AutoTokenizer.from_pretrained(model_id) recipe = QuantizationModifier( targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"],)oneshot(model=model, recipe=recipe) model.save_pretrained(output_dir, save_compressed=True)tokenizer.save_pretrained(output_dir)print(f"Saved the FP8 checkpoint to {output_dir}")
The saved compressed-tensors checkpoint includes the FP8 weights and their scales. Activations are not stored; vLLM calculates their scales and converts them to FP8 during each forward pass.
SmoothQuant can be added when activation outliers cause too much error. This makes the conversion calibration-based. The following complete script loads the same model and calibration dataset as the INT8 example, then saves a separate FP8 checkpoint:
Calibrate and save an FP8 checkpoint with SmoothQuant.
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.modifiers.transform.smoothquant import SmoothQuantModifier
model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
output_dir = "models/tinyllama-w8a8-fp8-smoothquant"
num_calibration_samples = 128
max_sequence_length = 512
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
dtype="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
dataset = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
dataset = dataset.shuffle(seed=42).select(range(num_calibration_samples))
def preprocess(example):
text = tokenizer.apply_chat_template(example["messages"], tokenize=False)
return {"text": text}
def tokenize(example):
return tokenizer(
example["text"],
max_length=max_sequence_length,
truncation=True,
add_special_tokens=False,
)
dataset = dataset.map(preprocess)
dataset = dataset.map(tokenize, remove_columns=dataset.column_names)
recipe = [
SmoothQuantModifier(smoothing_strength=0.8),
QuantizationModifier(
targets="Linear",
scheme="FP8_DYNAMIC",
ignore=["lm_head"],
),
]
oneshot(
model=model,
dataset=dataset,
recipe=recipe,
max_seq_length=max_sequence_length,
num_calibration_samples=num_calibration_samples,
)
model.save_pretrained(output_dir, save_compressed=True)
tokenizer.save_pretrained(output_dir)
print(f"Saved the FP8 checkpoint to {output_dir}")
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657
from datasets import load_datasetfrom transformers import AutoModelForCausalLM, AutoTokenizer from llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifierfrom llmcompressor.modifiers.transform.smoothquant import SmoothQuantModifier model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"output_dir = "models/tinyllama-w8a8-fp8-smoothquant"num_calibration_samples = 128max_sequence_length = 512 model = AutoModelForCausalLM.from_pretrained( model_id, device_map="auto", dtype="auto",)tokenizer = AutoTokenizer.from_pretrained(model_id) dataset = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")dataset = dataset.shuffle(seed=42).select(range(num_calibration_samples)) def preprocess(example): text = tokenizer.apply_chat_template(example["messages"], tokenize=False) return {"text": text} def tokenize(example): return tokenizer( example["text"], max_length=max_sequence_length, truncation=True, add_special_tokens=False, ) dataset = dataset.map(preprocess)dataset = dataset.map(tokenize, remove_columns=dataset.column_names) recipe = [ SmoothQuantModifier(smoothing_strength=0.8), QuantizationModifier( targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"], ),] oneshot( model=model, dataset=dataset, recipe=recipe, max_seq_length=max_sequence_length, num_calibration_samples=num_calibration_samples,) model.save_pretrained(output_dir, save_compressed=True)tokenizer.save_pretrained(output_dir)print(f"Saved the FP8 checkpoint to {output_dir}")
In both cases, you can load the local FP8 checkpoint and run inference as follows:
Load the saved FP8 checkpoint and generate text with vLLM.
from vllm import LLM, SamplingParams
model_path = "models/tinyllama-w8a8-fp8"
model = LLM(model=model_path)
sampling_params = SamplingParams(temperature=0, min_tokens=16, max_tokens=64)
outputs = model.generate(
["Explain FP8 quantization in one paragraph."],
sampling_params,
)
print(outputs[0].outputs[0].text)
123456789101112
from vllm import LLM, SamplingParams model_path = "models/tinyllama-w8a8-fp8"# Or use: models/tinyllama-w8a8-fp8-smoothquantmodel = LLM(model=model_path)sampling_params = SamplingParams(temperature=0, min_tokens=16, max_tokens=64) outputs = model.generate( ["Explain FP8 quantization in one paragraph."], sampling_params,)print(outputs[0].outputs[0].text)
The generated text only confirms that the saved checkpoint runs. Compare accuracy and throughput with the unquantized model before choosing a deployment format.
Current vLLM documentation lists NVIDIA Ada Lovelace, Hopper, and Blackwell GPUs, as well as some AMD accelerators, for accelerated FP8 weight-and-activation inference. The exact path depends on the GPU, FP8 layout, and quantization scheme, so verify the current compatibility table for the deployment hardware.
Activation quantization may improve matrix-heavy prefill, but its runtime scaling can add overhead during small-batch decode. A combined weight-and-activation scheme can move the bottleneck to another operation, so profile the full request rather than adding isolated speedups.
Further Reading #
Below are some resources you may find useful:
- PyTorch TorchAO documentation. Quantized Inference . This guide lists current per-token, per-row, per-tensor, and block-scaled activation configurations.
- Guangxuan Xiao et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models . This paper introduces mathematically equivalent smoothing for W8A8 inference.
- vLLM documentation. INT8 W8A8 with LLM Compressor and vLLM . This guide covers calibration, conversion, compressed checkpoint saving, and inference.
- Paulius Micikevicius et al. (2022). FP8 Formats for Deep Learning . This paper describes FP8 formats and scaling conventions for deep learning.
- vLLM documentation. FP8 weight-and-activation quantization with LLM Compressor and vLLM . This guide covers dynamic FP8 conversion, checkpoint , and supported hardware.
Summary #
In this post, you learned why activations are harder to quantize than weights: their ranges are input-dependent and often contain outliers. SmoothQuant moves part of this difficulty into static weights so W8A8 matrix multiplication becomes practical. FP8 provides floating-point range at eight bits, but still requires explicit format and scaling choices.
You also learned how to create reusable INT8 W8A8 and FP8 weight-and-activation checkpoints with LLM Compressor and run them with vLLM.