Demystifying Quantizations: Guide to Quantization Methods for LLMs

A technical guide from Cast AI explains that quantization, the process of compressing LLM weights to lower-precision data types, is central to balancing throughput, memory, and inference costs, and clarifies that GGUF is a file format, not a quantization algorithm. The guide highlights that GPTQ was the first 4-bit method, AWQ improves accuracy via activation-aware scaling and runs 4.96 seconds vs. 8.78 seconds for GPTQ on a Mistral 7B benchmark on an A100-80GB, and SmoothQuant enables W8A8 integer inference. It notes that a 70B model requiring ~140 GB in fp16 fits on a single A100-80GB when quantized to Q4_K_M (~42 GB), making quantization a practical infrastructure decision.

Selecting which LLM to deploy means balancing throughput, memory footprint, accuracy, and LLM inference costs https://cast.ai/blog/llm-inference-cost-optimization/ . Quantization sits at the center of that balance. It is the process of constraining a model’s values from a continuous or large set to a discrete, lower-precision set. The term “quantization” appears constantly alongside high-throughput serving engines like vLLM, SGLang, and Triton. Yet few resources make quantization methods approachable for practitioners who aren’t deep-learning researchers. Even the vLLM documentation lists GGUF under quantization options, which leads many readers to assume GGUF is a quantization algorithm. It is not. GGUF is a file format. This guide will make that distinction, and many others, clear. There is a well-known quote by Tim Dettmers that captures the essence of quantization research perfectly: “Quantization research is like printers. Nobody cares about printers. Nobody likes printers. But everybody is happy if printers do their job.” Tim Dettmers Key Takeaways - Quantization converts model weights and sometimes activations to lower-precision data types, reducing memory usage and, in many cases, improving inference speed. - Post-training quantization PTQ applies after training and is the practical default for open-source LLMs. Quantization-aware training QAT yields better accuracy at 4-bit and below but requires a full training run. - GPTQ W4A16 was the first method to compress LLMs to 4-bit precision. AWQ improves on it by applying activation-aware per-channel scaling to protect the most sensitive weights before quantization, resulting in better accuracy and faster inference. SmoothQuant enables full W8A8 integer inference by migrating quantization difficulty from activations to weights. - GGUF is a container file format, not a quantization algorithm. It packages weights, tokenizer, and metadata in one portable file. The weights inside a GGUF file can be quantized at various levels Q4 0, Q4 K M, Q5 K M, Q8 0 . - On a Mistral 7B benchmark on an A100-80GB, AWQ completes inference in 4.96 seconds vs. 8.78 seconds for GPTQ, a meaningful gap for latency-sensitive workloads. - Hardware support determines which quantization types actually accelerate inference. Not every GPU supports every format natively. - A 70B model that requires ~140 GB in fp16 fits on a single A100-80GB GPU when quantized to Q4 K M ~42 GB , making quantization a practical infrastructure decision, not just an accuracy trade-off. Why quantization? Recap of data types used in LLMs To understand quantization, start with a quick recap of the data types involved. When you download an open-source model, the neural network inside is essentially a collection of numbers stored across multiple files with accompanying metadata. The precision data type chosen for those numbers determines both model size and computational cost. Integers Integers are the most basic data type, represented as a sequence of bits. They are straightforward, efficient, and inexpensive to compute with, but they sacrifice precision. For example, representing a bank account balance solely with integers would be highly unreliable. The same trade-off applies when representing the weights of a large model. Floating-point number representation When you need to represent fractional values with high precision, floating-point number representation is the answer. The governing standard is IEEE 754 https://en.wikipedia.org/wiki/IEEE 754 . Single precision In a 32-bit representation: - 1 bit is reserved for the sign, allowing both positive and negative values. - 8 bits are allocated to the exponent, which defines the range of representable values: essentially how large or small a number can be. - 23 bits are used for the mantissa, which determines the level of precision. A 32-bit floating-point number single precision was the dominant format for training neural networks throughout most of deep learning history. Half precision The IEEE 754 standard also defines double precision 64-bit and half precision 16-bit . Double precision is not widely used in LLM deployment, so it is skipped here. Half precision has become the de facto standard for newly released models, with an important twist. Most recently published models do not follow the standard 16-bit format even when labeled as 16-bit. The twist: Google invented a format called bfloat brain floating point , which is now the standard for publishing unquantized models. bfloat The difference lies in how bf16 splits bits between exponent and mantissa. bfloat16 uses the same number of exponent bits as IEEE single-point precision fp32 , making conversion between bf16 and fp32 straightforward. It keeps the same dynamic range as fp32 but sacrifices precision in the mantissa. For LLM training and serving, this is a favorable trade-off: dynamic range matters more than fine-grained precision. 4-bit numbers Finally, consider 4-bit numbers such as int4 and fp4. Models can technically be quantized using binary or ternary schemes, but those fall outside the scope here. In practice, 4-bit precision is the lowest useful level applied in post-training quantization. Each format presents its own trade-offs. fp4 e1m2 prioritizes precision over dynamic range and can represent infinite or NaN values. fp4 e3m0 sacrifices that capability, making it unable to represent NaN or infinite values. The number of bits allocated to exponent vs. mantissa directly determines the range of values the format can handle. Model size impact is significant here. For the Qwen3-32B https://huggingface.co/Qwen/Qwen3-32B model, moving from fp16 to a 4-bit format means roughly 45 GB less memory needed. That difference determines which GPUs can run the model and how expensive inference becomes. Energy needed to execute per format With data types defined, it is worth examining the energy efficiency of different formats. The table below from Mark Horowitz’s Computing’s Energy Problem article https://gwern.net/doc/cs/hardware/2014-horowitz-2.pdf shows the energy required to perform specific operations depending on the numerical format used. The choice of number format and operation significantly impacts the efficiency of ML pipelines. Integer operations are substantially cheaper than floating-point operations of the same width. This is why reducing the precision of model weights translates directly into lower inference energy costs, not just lower memory usage. Intuition behind the neural network quantization: common types of values and operations Before diving into quantization algorithms, it helps to recall what a forward pass actually computes. The diagram below illustrates a single artificial neuron. Regardless of model architecture, each neuron applies weight adjustments followed by an activation function. The output is unbounded and can range from negative infinity to positive infinity. The sigma symbol in the diagram represents a placeholder for any general activation function, not specifically the sigmoid. Before the activation function applies, the output can span the entire real number range depending on inputs and weights. From this picture, the key components to keep in mind are: - Weights - Activations - Bias - Inputs Each quantization method handles these components differently when quantizing the network. Weights are quantized in all practical methods. Activation values are handled differently: some methods leave them in fp16, others quantize them too. That distinction drives most of the performance and accuracy differences between the approaches discussed later. Quantization: a short history Quantization as an idea did not originate in machine learning. It came from signal processing. Before 2017, quantization for neural networks was mostly an academic topic. Then the paper Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference https://arxiv.org/pdf/1712.05877 changed that. For the first time, researchers had solid evidence that they could apply quantization in production. TensorFlow Lite implemented the methods from that paper using a linear quantization approach. Two common methods were established in this era: - K-means-based quantization - Linear quantization Both methods originated well before the Transformer architecture. They were widely applied to convolutional neural networks CNNs , which at the time represented the primary commercial workload in deep learning. K-means-based quantization Weights in any given layer are typically normally distributed with a small number of outliers. The graph below shows the weight density of a pruned and fine-tuned model, where the distribution appears bimodal rather than normal. Values near zero were pruned, and the graph reproduces results from the Deep Compression https://arxiv.org/abs/1510.00149 paper. Once K-means quantization is applied, the weight distribution becomes discrete. Only a few centroids remain, as shown in the image below. In short: K-means clustering is applied to the input weights, finding 2<sup n</sup different centroids to map continuous values to discrete values. The number of bits used n determines how many centroids are available. Reducing the number of centroids reduces the precision data type’s expressiveness, but also reduces the memory footprint. Linear quantization Linear quantization is an affine mapping of integers to real numbers. There are two modes: symmetric and asymmetric. The asymmetric mode is illustrated in the image below. Weights are represented as real numbers r , and the objective is to map the continuous floating-point space to a small set of discrete integer values q . The min and max integer values depend on the bit-width chosen. For 4-bit quantization, integer values ranging from -8 to 7 can represent weights. The scaling factor S and zero point Z then control the mapping. For symmetric quantization, you compute the scaling factor as: S = max |x| / 2^ bits-1 - 1 Quantized value = round x / S + Z The objective is to minimize the squared error between the original and quantized values. During dequantization, the original value is approximated by reversing the mapping. The quantization error is the difference between the original value and that approximation. Comparing the original fp16 input vector blue and the reconstructed fp16 vector yellow , a visible difference appears. That difference is the quantization error. Minimizing it is what every subsequent quantization method is trying to do, each with its own approach. Quantization in the LLM era Why are the classical methods not good enough for quantizing LLMs? The graph below from the LLM.int8 paper https://arxiv.org/abs/2208.07339 shows that at around 6.7B parameters, certain features begin to emerge that break classical quantization assumptions. In standard neural networks, most weights look like this: 0.32, 0.64, 0.98, 0.11, 0.43 But beyond 6.7B parameters, some activation vectors look like this: -60, -45, -51, -35, -20, -67 Outliers like the one above usually occur in activations, but they can also occur in weights. In some literature, these vectors are called saliency weights, meaning they are the weights that contribute the most to the model’s output. Usual methods will not work well here. The outlier inside the weights or activations will make our methods squish and cause us to lose too much knowledge. Dynamic vs. static quantization - Dynamic quantization : The quantization range for activations is computed at runtime, per inference. This avoids the need for a calibration dataset but adds overhead per forward pass. - Static quantization : Ranges are pre-computed offline from a representative calibration dataset and fixed for inference. This is faster at runtime but requires a calibration step and can fail if the production distribution differs from calibration data. Most LLM-specific methods use static quantization for weights and either static or dynamic quantization for activations, depending on what the hardware and the method support. Note: There are two main approaches to quantization: post-training quantization PTQ and quantization-aware training QAT . This article focuses primarily on PTQ, which applies after the model finishes training. A brief comparison appears in the section below. Post-training quantization vs. quantization-aware training Post-training quantization PTQ applies after training is complete. It is the practical default for open-source LLMs: you download the model and quantize it. No training infrastructure required. Quantization-aware training QAT takes a different approach. During training, it simulates quantization using fake quantization operations so the model can adapt its weights to minimize quantization errors before they are actually applied. The result is better accuracy than PTQ, particularly at very low bit-widths like 4-bit and below. The cost is a full training run, which makes QAT significantly more compute-intensive and less accessible for most practitioners working with pre-trained open-source models. For most production LLM deployments today, PTQ methods GPTQ, AWQ, SmoothQuant are the practical choice. QAT becomes relevant when you control the training process and need to push accuracy at very aggressive quantization levels. Quantization methods for LLMs Warning: This article will not cover the following methods in depth. Instead, the focus is on highlighting their importance and key contributions to the field. Note on all ‘When to use it?’ sections: These provide only a rough idea. The actual choice depends on many more details and specific circumstances. GPTQ GPTQ was the first quantization method to compress LLMs down to the 4-bit range while maintaining usable accuracy. It uses second-order Hessian information to perform one-shot weight quantization layer by layer. The insight: rather than minimizing a simple reconstruction error, GPTQ minimizes the change in the layer’s output caused by quantizing its weights, using curvature information from the Hessian to make smarter rounding decisions. GPTQ operates in W4A16 mode: weights are quantized to 4-bit, while activation values remain in fp16. It does not quantize activations. At the time the paper was published, hardware did not yet provide native speedups for 4-bit inference. Since then, GPTQ speedups have become available on some hardware, and the method remains widely used in vLLM and TGI. When to use it: GPTQ provides significant memory savings with a well-understood accuracy trade-off. It is a reliable baseline for 4-bit weight quantization and is broadly supported across inference frameworks. SmoothQuant SmoothQuant solves a different problem. The challenge with quantizing both weights and activations to 8-bit W8A8 is that activations have far more outliers than weights. Squishing a large activation outlier into an 8-bit integer range causes significant accuracy loss. SmoothQuant’s solution: migrate the quantization difficulty from activations to weights via a per-channel scaling factor S. Concretely: - Divide the activation values by S smoothing them out, making them easier to quantize - Multiply the corresponding weights by S absorbing the difficulty into the weights, which are already smoother The result: both tensors are smooth enough to quantize to int8, enabling full integer matrix multiplication on both. This matters because hardware accelerators can execute integer matrix multiplication significantly faster than floating-point equivalents. W8A8 with SmoothQuant enables speedups that W4A16 methods like GPTQ cannot achieve for batch workloads, since the bottleneck shifts from memory bandwidth to compute. When to use it: SmoothQuant is most useful for batch inference workloads where compute throughput matters. SmoothQuant was tested on a 530B model running on a single 8-GPU node 8x A100-80GB – a configuration that required float16 previously. This demonstrates how W8A8 quantization can halve memory requirements while maintaining near-identical accuracy. Note that single-request latency may not improve significantly on memory-bandwidth-limited hardware, since the bottleneck for single inference is often memory, not arithmetic throughput. Activation aware quantization AWQ AWQ Activation-Aware Weight Quantization takes a more targeted approach to the outlier problem. Instead of smoothing the entire activation distribution, AWQ identifies the salient weight channels by examining activation magnitudes across a calibration dataset. Specifically, AWQ applies per-channel scaling factors to those salient channels before quantization. All weights are still quantized to 4-bit — the scaling factors protect accuracy without mixed precision by making the most important weights easier to quantize accurately. The key insight: AWQ scales up weights corresponding to large activation magnitudes before quantization, then divides the activations by the same scale factor during inference. This keeps the forward-pass output mathematically equivalent, but minimizes quantization error precisely where it matters most – on the weights the model relies on most heavily. Like GPTQ, AWQ operates in W4A16 mode: it stores all quantized weights in 4-bit and keeps activations in fp16. The distinction lies in how AWQ structures the quantization: it uses activation-aware per-channel scaling to protect salient weights rather than keeping any weights in a higher-precision format. This produces better accuracy on instruction-following tasks compared to GPTQ. The inference speed advantage is also notable. On a Mistral 7B benchmark on an A100 GPU, AWQ completes inference in 4.96 seconds vs. GPTQ at 8.78 seconds single-request inference benchmark on A100-80GB; source: AutoAWQ evaluation . That is a 44% reduction in inference time for the same model and hardware, driven by AWQ’s more efficient dequantization kernel design. When to use it: AWQ is the preferred choice when you need both memory savings and inference speed improvements. It outperforms GPTQ on instruction-following accuracy and is faster at runtime. It was published as a state-of-the-art method and remains highly competitive. GGUF GGUF is not a quantization method. This point deserves emphasis because blog posts, GitHub issues, and documentation consistently mislabel GGUF. GGUF is a file format , specifically the GGML Universal Format, which replaced the original GGML format in August 2023. A GGUF file packages everything needed to run a model: quantized weights, tokenizer, and metadata in a single portable file. The quantization happens separately, using methods implemented in the GGML library https://github.com/ggml-org/ggml , primarily by Ivan Kawrakow https://github.com/ikawrakow , who implemented most of the quantization techniques there in his spare time without publishing formal papers. The GGML library defines several block-based quantization approaches, computing a scaling factor per block of the tensor rather than per layer or globally. There are three generations of GGML quants: - Legacy quants Q4 0, Q4 1, Q5 0, etc. : Original format, still functional but not recommended for new deployments. - k-quants Q3 K S, Q4 K M, Q5 K M, Q6 K, etc. : Improved accuracy over legacy quants. Still widely used and a solid default choice. - i-quants IQ3 S, IQ4 NL, etc. : Current state-of-the-art in llama.cpp. Better accuracy-per-bit than k-quants. Recommended for new deployments. Common k-quant levels and their practical tradeoffs: | Quantization level | Bits per weight approx. | Quality | Use case | |---|---|---|---| | Q4 0 | 4.5 bpw | Baseline 4-bit legacy | Maximum compatibility, not recommended for quality | | Q4 K M | 4.8 bpw | Good 4-bit quality | Default 4-bit choice for most deployments | | Q5 K M | 5.7 bpw | Near-fp16 quality | When accuracy matters and memory allows | | Q8 0 | 8.5 bpw | Very close to fp16 | Maximum quality with quantization, largest footprint | GGUF is primarily used with llama.cpp and CPU-friendly inference. vLLM supports GGUF but does not have fully optimized kernels for all GGML quant types. For GPU-accelerated server inference, GPTQ and AWQ remain the more mature choices. When to use it: GGUF with quantized weights inside is ideal for local inference, edge deployments, and CPU-based or hybrid CPU-GPU setups. The k-quant and i-quant options give you a wide range of quality-size tradeoffs to explore. Start with Q4 K M and test against Q5 K M for your specific model and task. FP8 and hardware-native formats FP8 8-bit floating point has become increasingly important for H100 and H200 GPUs, which include hardware-accelerated FP8 matrix multiplication. Unlike integer quantization, FP8 retains the floating-point format but uses fewer bits — preserving dynamic range better than int8 while still cutting memory roughly in half compared to fp16. FP8 is now the default training precision for large models on Hopper architecture, and frameworks like vLLM and TensorRT-LLM increasingly use it for inference. It offers better accuracy than INT8 at similar memory efficiency, making it the format to watch for GPU-native serving in 2025 and beyond. FP8 exists in two variants. E4M3 4-bit exponent, 3-bit mantissa delivers higher dynamic range and is preferred for weights and KV cache. E5M2 5-bit exponent, 2-bit mantissa prioritizes range over precision and is better for gradient computation during training. For inference, E4M3 is the standard choice in vLLM and TensorRT-LLM. Accuracy impact is typically less than 0.5% perplexity degradation versus FP16 on most LLMs, making FP8 the lowest-friction quantization option available for H100 deployments. A related technique is KV cache quantization, which compresses the key-value attention cache during inference rather than the model weights. Tools like vLLM support FP8 KV cache quantization, which reduces memory pressure for long-context workloads without recompressing the model itself. Note about hardware Quantization is not purely a software decision. Each specific data type requires custom hardware implementation to deliver actual speedups. The hardware available to you determines which quantization types accelerate inference and which simply reduce memory usage without improving throughput. The vLLM quantization hardware support table https://docs.vllm.ai/en/stable/ is the clearest reference for which combinations of format and GPU actually work in practice. Not every GPU supports every quantized format with native kernel acceleration. To make this concrete, here is how GPU memory requirements change across quantization levels for commonly deployed large models: | Model | fp16 baseline | int8 W8 | Q4 K M 4-bit | GPU fit | |---|---|---|---|---| | 7B models e.g., Mistral 7B | ~14 GB | ~7 GB | ~4 GB | RTX 4090 24 GB handles fp16 or Q4 easily | | Qwen3-32B | ~64 GB | ~32 GB | ~20 GB | Q4 fits on a single A100-40GB | | Llama 3 70B | ~140 GB | ~70 GB | ~42 GB | Q4 K M fits on 1x A100-80GB | These numbers illustrate why quantization is often a practical infrastructure decision before it is an accuracy decision. A Llama 3 70B model in fp16 requires at least 4x A100-40GB GPUs weights alone, tight fit or 8x for production inference headroom with KV cache and batching. The same model at Q4 K M fits on a single A100-80GB. For teams managing GPU autoscaling https://cast.ai/blog/kubernetes-gpu-autoscaling/ and cost-sensitive AI inference at scale, this difference is significant. Conclusion This guide has covered LLM quantization from its foundations in data types and linear quantization through to the LLM-specific methods that power modern inference deployments. The core ideas: - Lower precision data types reduce memory and energy costs, but they can introduce quantization errors that require careful management. - Classical methods of quantization K-means, linear worked well for smaller neural networks but break down at LLM scale due to activation outliers. - GPTQ, SmoothQuant, and AWQ each address the outlier problem differently. GPTQ and AWQ use W4A16; SmoothQuant enables W8A8 integer matrix multiplication. - GGUF is a file format, not a quantization algorithm. The quantization in a GGUF file comes from GGML’s block-based methods. - Hardware determines which quantization types actually accelerate inference, not just reduce memory. For teams with H100 or H200 GPUs, FP8 has emerged as a low-friction path to quantization with minimal accuracy tradeoff – E4M3 in particular delivers sub-0.5% perplexity impact versus FP16, with no calibration dataset required for most inference frameworks. Cast AI https://cast.ai/llm-optimization/ automates GPU rightsizing, Spot instance adoption, and bin-packing for inference pods – independently of which quantization format you choose. Whether you deploy AWQ, GPTQ, or FP8 on a shared H100 fleet, Cast AI ensures those GPUs stay right-sized as traffic patterns shift. Average GPU utilization in production AI clusters is under 5%. Quantization makes your model smaller; Cast AI makes your GPU budget match your actual workload. Frequently Asked Questions What are quantization methods for LLMs? Quantization methods for LLMs are techniques that reduce the numerical precision of model weights and sometimes activations to decrease memory usage and improve inference speed. The most widely used methods today are GPTQ, AWQ, and SmoothQuant. Each takes a different approach to managing the outlier activations that appear in large models beyond ~6.7B parameters. What is the difference between GPTQ, AWQ, and SmoothQuant? GPTQ uses second-order Hessian information to quantize weights to 4-bit W4A16 , leaving activations in fp16. AWQ also produces W4A16 but uses activation-aware per-channel scaling to protect the most sensitive weight channels before quantization — all weights are still quantized to 4-bit, but the scaling minimizes quantization error where it matters most, resulting in better accuracy and faster inference than GPTQ. SmoothQuant enables W8A8 quantization both weights and activations in int8 by migrating the quantization difficulty from activations to weights via a per-channel scaling factor, enabling faster integer matrix multiplication on compatible hardware. Is GGUF a quantization method? No. GGUF is a file format GGML Universal Format that replaced GGML in August 2023. It packages model weights, tokenizer, and metadata in a single portable file. The weights stored inside a GGUF file can be quantized using GGML’s block-based methods Q4 0, Q4 K M, Q5 K M, Q8 0, etc. , but GGUF itself is the container, not the quantization algorithm. What is post-training quantization PTQ ? Post-training quantization applies quantization after the model has been fully trained. It does not require access to training infrastructure and works directly on the pretrained weights. PTQ is the standard approach for quantizing open-source LLMs. Methods like GPTQ, AWQ, and SmoothQuant are all PTQ methods. The trade-off compared to QAT is that PTQ produces slightly lower accuracy at very aggressive bit-widths 4-bit and below . What is quantization-aware training QAT ? Quantization-aware training simulates the effect of quantization during the training process using fake quantization operations. The model adapts its weights to minimize quantization errors before they are actually applied, resulting in better accuracy than PTQ, especially at 4-bit and below. The cost is a full training run, making QAT significantly more compute-intensive than PTQ. QAT is most relevant when you control the training pipeline and need to maximize accuracy at aggressive quantization levels. How does quantization affect inference speed? Quantization can significantly improve inference speed, but the effect depends on both the method and the hardware. On a Mistral 7B single-request inference benchmark using an A100-80GB GPU, AWQ completes inference in 4.96 seconds vs. 8.78 seconds for GPTQ source: AutoAWQ evaluation; throughput at batch size may differ . SmoothQuant enables W8A8 integer matrix multiplication, which can accelerate batch inference on hardware with strong int8 support like the A100 and H100 . Hardware must natively support a given quantization format for speedups to materialize; otherwise, quantization only reduces memory usage. How much GPU memory does a quantized LLM need? Memory requirements depend on model size and quantization level. A 7B model requires ~14 GB in fp16, ~7 GB in int8, and ~4 GB in Q4. A 32B model e.g., Qwen3-32B requires ~64 GB in fp16, ~32 GB in int8, and ~20 GB in Q4, fitting on a single A100-40GB. A 70B model e.g., Llama 3 70B requires ~140 GB in fp16 needing 4x A100-40GB at minimum for weights, or 8x for production headroom but only ~42 GB at Q4 K M, fitting on a single A100-80GB. When should I use 4-bit vs. 8-bit quantization? Use 4-bit quantization GPTQ, AWQ, Q4 K M when memory is the primary constraint and you need to fit a large model on fewer GPUs. 4-bit reduces model size by ~4x compared to fp16 but introduces more quantization error. Use 8-bit quantization SmoothQuant W8A8, Q8 0 when you want near-fp16 accuracy with meaningful memory savings, or when you need integer matrix multiplication throughput for batch workloads on hardware with strong int8 support. For latency-sensitive single-request inference, AWQ 4-bit often outperforms 8-bit methods due to reduced memory bandwidth pressure.