Activation and Floating-Point Quantization A Machine Learning Mastery tutorial explains why activation tensors created during inference are harder to quantize than fixed weights: activation ranges depend on the input token, sequence position and previous layers, and can shift from request to request, with a few channels carrying persistent outliers. The post contrasts static activation quantizers, which fix scales during calibration and risk clipping inputs outside the calibration range, with dynamic per-token quantizers that compute a scale at runtime from the absolute maximum along the hidden dimension, and it covers how W8A8 methods such as SmoothQuant handle activation ranges and how floating-point quantization differs from integer quantization. Weight-Only Quantization https://machinelearningmastery.com/weight-only-quantization/ focused on weights, which are fixed after training and can be quantized once before deployment. This post turns to tensors created during inference. Activations depend on the input token, sequence position, and previous layers. Their ranges can change from request to request, and a few channels may contain persistent outliers. This makes activation quantization harder than weight-only quantization. In this post, you will learn about: - Why dynamic activations are harder to quantize than fixed weights - How W8A8 methods handle activation ranges - How floating-point quantization differs from integer quantization Let’s get started. Overview This post is divided into three parts; they are: - Why Activations Are Hard to Quantize - W8A8 Inference and SmoothQuant - Floating-Point Quantization Why Activations Are Hard to Quantize The post on weight-only quantization https://machinelearningmastery.com/weight-only-quantization/ introduced scales, zero points, and quantization granularity for fixed weights. The same concepts apply to activations, but the quantization parameters may now need to follow values that do not exist until inference begins. Consider an activation tensor entering a linear layer. A static activation quantizer uses scales chosen during calibration. It is fast because the scales are already known, but a deployment input outside the calibration range may be clipped. A dynamic activation quantizer measures the tensor at runtime. It adapts to each input but adds reduction and scaling work to the inference path. An activation quantizer can be written as follows: python Quantize an activation tensor with scales computed at runtime. import torch def quantize activation per token x : qmax = 127 scale = x.abs .amax dim=-1, keepdim=True / qmax scale = scale.clamp min torch.finfo x.dtype .eps q = torch.round x / scale .clamp -qmax, qmax return q.to torch.int8 , scale x = torch.randn 2, 32, 1024 q, scale = quantize activation per token x x hat = q.float scale print "mean absolute error:", x - x hat .abs .mean .item 12345678910111213 import torch def quantize activation per token x : qmax = 127 scale = x.abs .amax dim=-1, keepdim=True / qmax scale = scale.clamp min torch.finfo x.dtype .eps q = torch.round x / scale .clamp -qmax, qmax return q.to torch.int8 , scale x = torch.randn 2, 32, 1024 q, scale = quantize activation per token x x hat = q.float scaleprint "mean absolute error:", x - x hat .abs .mean .item For an activation shaped batch, sequence, hidden , each vector along the final dimension represents one token. Reducing over that dimension produces a scale shaped batch, sequence, 1 : one scale shared by all hidden features of each token. A linear layer treats these token vectors as matrix rows, so this scheme is also called per-row quantization. Dynamic per-token scaling is a common implementation for transformer activations. It follows token-to-token range changes more closely than one scale for the entire tensor. The example uses signed symmetric INT8, so the zero point is fixed at zero instead of being calculated for every token. This simplifies the quantization and matrix-multiplication arithmetic. The trade-off is that the absolute-maximum reduction and scale calculation must run whenever a new activation tensor is produced. Transformer activations often have outlier values in the vector. Covering the entire range would make the step size coarse for ordinary values. Clipping improves resolution for ordinary values but destroys the outlier. The best choice depends on how the following weights use that vector. Activation distributions also differ by layer and token. Static activation quantization uses calibration data to predict the ranges of future runtime tensors, then select the most appropriate scale and zero point. Dynamic activation quantization avoids fixed calibration ranges but pays the runtime measurement cost. W8A8 Inference and SmoothQuant Here, W8A8 means INT8 weights and INT8 activations for matrix multiplication. When the hardware provides efficient integer matrix multiplication, this can reduce memory traffic and increase arithmetic throughput. Biases, normalization, residuals, or accumulators may remain at higher precision. W8A8 by itself names bit widths; another workflow could use an FP8 format. SmoothQuant addresses activation outliers by moving some quantization difficulty from activations into weights. For a linear layer $\mathbf{y}=\mathbf{x}\mathbf{W}^\top$, choose a positive scale vector $\mathbf{s}$ over input channels: $$ \mathbf{y} = \mathbf{x} \operatorname{diag} \mathbf{s} ^{-1} \mathbf{W} \operatorname{diag} \mathbf{s} ^\top $$ The transformation is mathematically equivalent before quantization. Large activation channels are divided by larger scales, making the activation range smoother. The corresponding weight columns are multiplied by the same scales. Weights are static and usually tolerate this uneven range better with per-channel quantization. This differs from AWQ in the previous post https://machinelearningmastery.com/weight-only-quantization/ . AWQ uses calibration activations to protect important weights in a weight-only format such as W4A16; the runtime activations remain in a higher-precision format. SmoothQuant prepares both sides for W8A8 execution: it changes the stored weights and makes the runtime activations easier to quantize to INT8. The following code checks the equivalence: python Check the algebraic equivalence used by SmoothQuant. import torch import torch.nn.functional as F x = torch.randn 4, 128 weight = torch.randn 256, 128 smooth scale = torch.rand 128 + 0.5 reference = F.linear x, weight smoothed x = x / smooth scale smoothed weight = weight smooth scale transformed = F.linear smoothed x, smoothed weight torch.testing.assert close transformed, reference, rtol=1e-5, atol=1e-5 12345678910111213 import torchimport torch.nn.functional as F x = torch.randn 4, 128 weight = torch.randn 256, 128 smooth scale = torch.rand 128 + 0.5 reference = F.linear x, weight smoothed x = x / smooth scalesmoothed weight = weight smooth scaletransformed = F.linear smoothed x, smoothed weight torch.testing.assert close transformed, reference, rtol=1e-5, atol=1e-5 This only demonstrates the exact rescaling. A SmoothQuant implementation derives smooth scale from weight magnitudes and calibration activations, folds scales into adjacent operations where possible, and quantizes the transformed tensors. One practical way to create a reusable INT8 weight-and-activation checkpoint is LLM Compressor, a library in the vLLM project. The following workflow uses transformers to load the model, LLM Compressor to apply SmoothQuant and INT8 W8A8 quantization, and then saves the result for vLLM inference: Calibrate and save an INT8 W8A8 checkpoint with LLM Compressor. from datasets import load dataset from transformers import AutoModelForCausalLM, AutoTokenizer from llmcompressor import oneshot from llmcompressor.modifiers.gptq import GPTQModifier from llmcompressor.modifiers.transform.smoothquant import SmoothQuantModifier model id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" output dir = "models/tinyllama-w8a8" num calibration samples = 128 max sequence length = 512 model = AutoModelForCausalLM.from pretrained model id, device map="auto", dtype="auto", tokenizer = AutoTokenizer.from pretrained model id dataset = load dataset "HuggingFaceH4/ultrachat 200k", split="train sft", dataset = dataset.shuffle seed=42 .select range num calibration samples def preprocess example : text = tokenizer.apply chat template example "messages" , tokenize=False return {"text": text} def tokenize example : return tokenizer example "text" , max length=max sequence length, truncation=True, add special tokens=False, dataset = dataset.map preprocess dataset = dataset.map tokenize, remove columns=dataset.column names recipe = SmoothQuantModifier smoothing strength=0.8 , GPTQModifier targets="Linear", scheme="W8A8", ignore= "lm head" , oneshot model=model, dataset=dataset, recipe=recipe, max seq length=max sequence length, num calibration samples=num calibration samples, model.save pretrained output dir, save compressed=True tokenizer.save pretrained output dir print f"Saved the INT8 checkpoint to {output dir}" 1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859 from datasets import load datasetfrom transformers import AutoModelForCausalLM, AutoTokenizer from llmcompressor import oneshotfrom llmcompressor.modifiers.gptq import GPTQModifierfrom llmcompressor.modifiers.transform.smoothquant import SmoothQuantModifier model id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"output dir = "models/tinyllama-w8a8"num calibration samples = 128max sequence length = 512 model = AutoModelForCausalLM.from pretrained model id, device map="auto", dtype="auto", tokenizer = AutoTokenizer.from pretrained model id dataset = load dataset "HuggingFaceH4/ultrachat 200k", split="train sft", dataset = dataset.shuffle seed=42 .select range num calibration samples def preprocess example : text = tokenizer.apply chat template example "messages" , tokenize=False return {"text": text} def tokenize example : return tokenizer example "text" , max length=max sequence length, truncation=True, add special tokens=False, dataset = dataset.map preprocess dataset = dataset.map tokenize, remove columns=dataset.column names recipe = SmoothQuantModifier smoothing strength=0.8 , GPTQModifier targets="Linear", scheme="W8A8", ignore= "lm head" , oneshot model=model, dataset=dataset, recipe=recipe, max seq length=max sequence length, num calibration samples=num calibration samples, model.save pretrained output dir, save compressed=True tokenizer.save pretrained output dir print f"Saved the INT8 checkpoint to {output dir}" The saved checkpoint contains per-channel INT8 weights and instructions for dynamically quantizing activations during inference. The activations themselves are not stored in the checkpoint because they are produced and quantized per token at runtime. vLLM can load the local checkpoint and use its W8A8 kernels without a separate quantization argument: Load the saved INT8 W8A8 checkpoint and generate text with vLLM. from vllm import LLM, SamplingParams model = LLM model="models/tinyllama-w8a8" sampling params = SamplingParams temperature=0, min tokens=16, max tokens=64 outputs = model.generate "Explain activation quantization in one paragraph." , sampling params, print outputs 0 .outputs 0 .text 12345678910 from vllm import LLM, SamplingParams model = LLM model="models/tinyllama-w8a8" sampling params = SamplingParams temperature=0, min tokens=16, max tokens=64 outputs = model.generate "Explain activation quantization in one paragraph." , sampling params, print outputs 0 .outputs 0 .text This short generation checks that the checkpoint loads and runs. The small model’s answer may be inaccurate; judge quantization quality against the original model on representative evaluation prompts. The checkpoint format, model architecture, vLLM version, and GPU must support the same W8A8 scheme. Current vLLM INT8 kernels target NVIDIA GPUs from Turing through Hopper; Blackwell users should choose a supported FP8 path instead. W8A8 is not guaranteed to be faster than W4A16 or BF16. W8A8 can use integer matrix hardware efficiently, especially for prefill and larger batches. W4A16 reads fewer weight bytes and can be attractive for small-batch decode. You should benchmark both phases. Floating-Point Quantization FP8 quantization does not simply crop bits from an FP32 value. In a common scaled FP8 scheme, the original tensor is divided by a scale and each result is rounded to a representable FP8 value: $$ \begin{aligned} q &= \operatorname{cast} {\text{FP8}} x/s , \\ \hat{x} &= s\operatorname{cast} {\text{FP16 or FP32}} q \end{aligned} $$ The scale must accompany the FP8 tensor or block so the runtime can interpret its values. FP8 is therefore still a quantized representation rather than an ordinary unscaled cast to a smaller storage type. Unlike affine INT8, FP8 does not place representable values at one regular step size across the whole quantization group. Its exponent gives it nonuniform spacing: values near zero have finer steps, while larger values have wider steps. This covers a broader range of magnitudes, although rounding and overflow are still possible. Scaling places the source values in the most useful part of that FP8 range; it normally does not require an INT8-style zero-point shift. The word “FP8” does not specify which FP8 layout is used. For example, E4M3 has four exponent bits and three significand bits, while E5M2 has five exponent bits and two significand bits. E4M3 provides more precision, whereas E5M2 provides a wider range. Which one can be used depends on the hardware and inference engine. FP8 also does not specify whether scaling is per tensor or per block, or how scales are calculated and stored. FP4 is similar, but requires more aggressive scaling and appropriate hardware support. The vLLM project’s LLM Compressor can convert a Hugging Face checkpoint to an FP8 checkpoint. Its FP8 DYNAMIC scheme uses round-to-nearest quantization for the fixed weights and derives their scales directly from the weight tensors. Activations are not available during conversion, so vLLM measures them and calculates per-token scales during inference. Neither operation needs representative input samples, which is why this basic workflow does not provide calibration data: python Save a dynamically quantized FP8 checkpoint. from transformers import AutoModelForCausalLM, AutoTokenizer from llmcompressor import oneshot from llmcompressor.modifiers.quantization import QuantizationModifier model id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" output dir = "models/tinyllama-w8a8-fp8" model = AutoModelForCausalLM.from pretrained model id, dtype="auto" tokenizer = AutoTokenizer.from pretrained model id recipe = QuantizationModifier targets="Linear", scheme="FP8 DYNAMIC", ignore= "lm head" , oneshot model=model, recipe=recipe model.save pretrained output dir, save compressed=True tokenizer.save pretrained output dir print f"Saved the FP8 checkpoint to {output dir}" 123456789101112131415161718192021 from transformers import AutoModelForCausalLM, AutoTokenizer from llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifier model id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"output dir = "models/tinyllama-w8a8-fp8" model = AutoModelForCausalLM.from pretrained model id, dtype="auto" tokenizer = AutoTokenizer.from pretrained model id recipe = QuantizationModifier targets="Linear", scheme="FP8 DYNAMIC", ignore= "lm head" , oneshot model=model, recipe=recipe model.save pretrained output dir, save compressed=True tokenizer.save pretrained output dir print f"Saved the FP8 checkpoint to {output dir}" The saved compressed-tensors checkpoint includes the FP8 weights and their scales. Activations are not stored; vLLM calculates their scales and converts them to FP8 during each forward pass. SmoothQuant can be added when activation outliers cause too much error. This makes the conversion calibration-based. The following complete script loads the same model and calibration dataset as the INT8 example, then saves a separate FP8 checkpoint: python Calibrate and save an FP8 checkpoint with SmoothQuant. from datasets import load dataset from transformers import AutoModelForCausalLM, AutoTokenizer from llmcompressor import oneshot from llmcompressor.modifiers.quantization import QuantizationModifier from llmcompressor.modifiers.transform.smoothquant import SmoothQuantModifier model id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" output dir = "models/tinyllama-w8a8-fp8-smoothquant" num calibration samples = 128 max sequence length = 512 model = AutoModelForCausalLM.from pretrained model id, device map="auto", dtype="auto", tokenizer = AutoTokenizer.from pretrained model id dataset = load dataset "HuggingFaceH4/ultrachat 200k", split="train sft" dataset = dataset.shuffle seed=42 .select range num calibration samples def preprocess example : text = tokenizer.apply chat template example "messages" , tokenize=False return {"text": text} def tokenize example : return tokenizer example "text" , max length=max sequence length, truncation=True, add special tokens=False, dataset = dataset.map preprocess dataset = dataset.map tokenize, remove columns=dataset.column names recipe = SmoothQuantModifier smoothing strength=0.8 , QuantizationModifier targets="Linear", scheme="FP8 DYNAMIC", ignore= "lm head" , , oneshot model=model, dataset=dataset, recipe=recipe, max seq length=max sequence length, num calibration samples=num calibration samples, model.save pretrained output dir, save compressed=True tokenizer.save pretrained output dir print f"Saved the FP8 checkpoint to {output dir}" 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657 from datasets import load datasetfrom transformers import AutoModelForCausalLM, AutoTokenizer from llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifierfrom llmcompressor.modifiers.transform.smoothquant import SmoothQuantModifier model id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"output dir = "models/tinyllama-w8a8-fp8-smoothquant"num calibration samples = 128max sequence length = 512 model = AutoModelForCausalLM.from pretrained model id, device map="auto", dtype="auto", tokenizer = AutoTokenizer.from pretrained model id dataset = load dataset "HuggingFaceH4/ultrachat 200k", split="train sft" dataset = dataset.shuffle seed=42 .select range num calibration samples def preprocess example : text = tokenizer.apply chat template example "messages" , tokenize=False return {"text": text} def tokenize example : return tokenizer example "text" , max length=max sequence length, truncation=True, add special tokens=False, dataset = dataset.map preprocess dataset = dataset.map tokenize, remove columns=dataset.column names recipe = SmoothQuantModifier smoothing strength=0.8 , QuantizationModifier targets="Linear", scheme="FP8 DYNAMIC", ignore= "lm head" , , oneshot model=model, dataset=dataset, recipe=recipe, max seq length=max sequence length, num calibration samples=num calibration samples, model.save pretrained output dir, save compressed=True tokenizer.save pretrained output dir print f"Saved the FP8 checkpoint to {output dir}" In both cases, you can load the local FP8 checkpoint and run inference as follows: python Load the saved FP8 checkpoint and generate text with vLLM. from vllm import LLM, SamplingParams model path = "models/tinyllama-w8a8-fp8" Or use: models/tinyllama-w8a8-fp8-smoothquant model = LLM model=model path sampling params = SamplingParams temperature=0, min tokens=16, max tokens=64 outputs = model.generate "Explain FP8 quantization in one paragraph." , sampling params, print outputs 0 .outputs 0 .text 123456789101112 from vllm import LLM, SamplingParams model path = "models/tinyllama-w8a8-fp8" Or use: models/tinyllama-w8a8-fp8-smoothquantmodel = LLM model=model path sampling params = SamplingParams temperature=0, min tokens=16, max tokens=64 outputs = model.generate "Explain FP8 quantization in one paragraph." , sampling params, print outputs 0 .outputs 0 .text The generated text only confirms that the saved checkpoint runs. Compare accuracy and throughput with the unquantized model before choosing a deployment format. Current vLLM documentation lists NVIDIA Ada Lovelace, Hopper, and Blackwell GPUs, as well as some AMD accelerators, for accelerated FP8 weight-and-activation inference. The exact path depends on the GPU, FP8 layout, and quantization scheme, so verify the current compatibility table for the deployment hardware. Activation quantization may improve matrix-heavy prefill, but its runtime scaling can add overhead during small-batch decode. A combined weight-and-activation scheme can move the bottleneck to another operation, so profile the full request rather than adding isolated speedups. Further Reading Below are some resources you may find useful: - PyTorch TorchAO documentation. Quantized Inference https://docs.pytorch.org/ao/stable/workflows/inference.html . This guide lists current per-token, per-row, per-tensor, and block-scaled activation configurations. - Guangxuan Xiao et al. 2022 . SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models https://arxiv.org/abs/2211.10438 . This paper introduces mathematically equivalent smoothing for W8A8 inference. - vLLM documentation. INT8 W8A8 with LLM Compressor and vLLM https://docs.vllm.ai/en/latest/features/quantization/llm compressor/int8 w8a8/ . This guide covers calibration, conversion, compressed checkpoint saving, and inference. - Paulius Micikevicius et al. 2022 . FP8 Formats for Deep Learning https://arxiv.org/abs/2209.05433 . This paper describes FP8 formats and scaling conventions for deep learning. - vLLM documentation. FP8 weight-and-activation quantization with LLM Compressor and vLLM https://docs.vllm.ai/en/latest/features/quantization/llm compressor/fp8/ . This guide covers dynamic FP8 conversion, checkpoint loading, and supported hardware. Summary In this post, you learned why activations are harder to quantize than weights: their ranges are input-dependent and often contain outliers. SmoothQuant moves part of this difficulty into static weights so W8A8 matrix multiplication becomes practical. FP8 provides floating-point range at eight bits, but still requires explicit format and scaling choices. You also learned how to create reusable INT8 W8A8 and FP8 weight-and-activation checkpoints with LLM Compressor and run them with vLLM.