{"slug": "weight-only-quantization", "title": "Weight-Only Quantization", "summary": "A technical tutorial published on Machine Learning Mastery details weight-only quantization, a technique that stores transformer weights as integers while keeping activations in FP16 or BF16 to cut the memory traffic that dominates autoregressive decoding at small batch sizes. The post walks through asymmetric affine quantization with scale and zero point, post-training quantization in PyTorch using TorchAO (installed via `python -m pip install torchao transformers accelerate`), GPTQ and AWQ, GGUF conversion through llama.cpp, and how to evaluate the memory, speed, and quality effects. It notes an INT8 weight uses half the raw storage of an FP16 or BF16 weight, and that low-precision matrix speedups depend on hardware, kernels, and model shapes.", "body_md": "During autoregressive decoding, a transformer reads its weights for every new token. At small batch sizes, moving these weights can take more time than computation. Quantization reduces this cost by storing weights with fewer bits. For example, an INT8 weight uses half the raw storage of an FP16 or BF16 weight.\n\nQuantization can also make computation faster. Many AI GPUs can perform low-precision matrix operations faster than FP16 operations. The actual speedup depends on the hardware, kernels, and model shapes.\n\nWeight-only quantization stores selected weights as integers while keeping activations in FP16 or BF16. This post begins with the asymmetric affine mapping, then shows how to quantize and run models with PyTorch and other inference tools.\n\nIn this post, you will learn about:\n\n- How weight-only quantization stores and reconstructs model weights\n- How post-training methods and GGUF conversion produce deployable artifacts\n- How to check the memory, speed, and quality effects of quantization\n\nLet’s get started.\n\n## Overview\n\nThis post is divided into five parts; they are:\n\n- Asymmetric Affine Quantization\n- Post-Training Quantization with PyTorch\n- GPTQ and AWQ\n- Quantizing a Model to GGUF\n- Evaluating Weight-Only Quantization\n\n**Note:** The code examples in this post use TorchAO and require a CUDA-enabled PyTorch installation. You can install the dependencies with:\n\n`python -m pip install torchao transformers accelerate`\n\nThe GPTQ and AWQ sections have their own installation steps. GGUF conversion requires a built `llama.cpp` checkout.\n\n## Asymmetric Affine Quantization\n\nInteger quantization maps a range of real values onto a finite set of integer codes. An asymmetric affine quantizer uses a scale $s$ and a zero point $z$:\n\n$$\n\nq = \\operatorname{clamp}\\Big(\\operatorname{round}\\big(\\frac{x}{s}\\big) + z, q_{\\min}, q_{\\max}\\Big)\n\n$$\n\nDequantization reconstructs an approximate floating-point value:\n\n$$\n\n\\hat{x} = s(q-z)\n\n$$\n\nFor an unsigned $b$-bit range, $q_{\\min}=0$ and $q_{\\max}=2^b-1$. Given the minimum and maximum values to represent, one common choice is:\n\n$$\n\n\\begin{aligned}\n\ns &= \\frac{x_{\\max}-x_{\\min}}{q_{\\max}-q_{\\min}} \\\\\n\nz &= \\operatorname{clamp}\\Big(\\operatorname{round}\\big(q_{\\min}-\\frac{x_{\\min}}{s}\\big),q_{\\min},q_{\\max}\\Big)\n\n\\end{aligned}\n\n$$\n\nThe scale determines the distance between adjacent representable values. The zero point is the integer code for real zero. Including zero in the real range ensures that it can be represented exactly.\n\n```\nQuantize and reconstruct a tensor to show the error from integer rounding.\n\nimport torch\n\ndef quantize_asymmetric(x, bits=8):\n    if not 1 <= bits <= 8:\n        raise ValueError(\"this uint8 example supports 1 to 8 bits\")\n    if not x.is_floating_point():\n        raise TypeError(\"x must be floating point\")\n    # max and min of quantized (unsigned int) values\n    qmin = 0\n    qmax = 2**bits - 1\n\n    # max and min of actual (float) values\n    xmin = torch.minimum(x.amin(), x.new_zeros(()))\n    xmax = torch.maximum(x.amax(), x.new_zeros(()))\n\n    # compute the scale and zero point, then quantize the actual values\n    scale = (xmax - xmin) / (qmax - qmin)\n    scale = scale.clamp_min(torch.finfo(x.dtype).eps)\n    zero_point = torch.round(qmin - xmin / scale).clamp(qmin, qmax)\n\n    q = torch.round(x / scale + zero_point).clamp(qmin, qmax)\n    return q.to(torch.uint8), scale, zero_point.to(torch.int32)\n\ndef dequantize(q, scale, zero_point):\n    return scale * (q.float() - zero_point.float())\n\nx = torch.tensor([-2.3, -0.2, 0.0, 0.7, 3.1])\nq, scale, zero_point = quantize_asymmetric(x)\nx_hat = dequantize(q, scale, zero_point)\n\nprint(\"scale:\", scale.item())\nprint(\"zero point:\", zero_point.item())\nprint(\"quantized:\", q)\nprint(\"maximum error:\", (x - x_hat).abs().max().item())\n\n12345678910111213141516171819202122232425262728293031323334\n\nimport torch def quantize_asymmetric(x, bits=8):    if not 1 <= bits <= 8:        raise ValueError(\"this uint8 example supports 1 to 8 bits\")    if not x.is_floating_point():        raise TypeError(\"x must be floating point\")    # max and min of quantized (unsigned int) values    qmin = 0    qmax = 2**bits - 1     # max and min of actual (float) values    xmin = torch.minimum(x.amin(), x.new_zeros(()))    xmax = torch.maximum(x.amax(), x.new_zeros(()))     # compute the scale and zero point, then quantize the actual values    scale = (xmax - xmin) / (qmax - qmin)    scale = scale.clamp_min(torch.finfo(x.dtype).eps)    zero_point = torch.round(qmin - xmin / scale).clamp(qmin, qmax)     q = torch.round(x / scale + zero_point).clamp(qmin, qmax)    return q.to(torch.uint8), scale, zero_point.to(torch.int32) def dequantize(q, scale, zero_point):    return scale * (q.float() - zero_point.float()) x = torch.tensor([-2.3, -0.2, 0.0, 0.7, 3.1])q, scale, zero_point = quantize_asymmetric(x)x_hat = dequantize(q, scale, zero_point) print(\"scale:\", scale.item())print(\"zero point:\", zero_point.item())print(\"quantized:\", q)print(\"maximum error:\", (x - x_hat).abs().max().item())\n```\n\nThe example demonstrates the numerical mapping, not efficient storage. A `uint8` tensor always uses one byte per element, even if `bits` is less than eight. A deployment format must pack sub-byte values to obtain the expected storage reduction. This example uses one scale and zero point for the whole tensor. Per-channel and per-group quantization often preserve LLM weight quality better because each channel or group gets its own range.\n\nFor a linear layer $\\mathbf{y}=\\mathbf{x}\\mathbf{W}^\\top$, weight-only quantization stores an integer matrix $\\mathbf{Q}$ plus scales and zero points instead of $\\mathbf{W}$:\n\n$$\n\n\\mathbf{y} \\approx \\mathbf{x}\\big(s(\\mathbf{Q}-z)\\big)^\\top\n\n$$\n\nThe input remains floating point. An efficient kernel would use the quantized weights directly instead of reconstructing the floating-point copy of $\\mathbf{W}$ like the equation shows.\n\nThe asymmetric affine mapping is the explanatory model used in this post. Production kernels and checkpoint formats may use a fixed zero point or another encoding. The runtime must support the exact quantization scheme and packed layout stored in the checkpoint.\n\n## Post-Training Quantization with PyTorch\n\nPyTorch’s current quantization development is centered in TorchAO. Its `quantize_()` function replaces the weights of matching `torch.nn.Linear` modules in place. Applying it to an evaluated, trained model is post-training quantization: no training step is performed.\n\nThe following example requests asymmetric, per-channel INT8 weight-only quantization. The activations remain BF16:\n\n``` python\nApply per-channel INT8 weight-only quantization with TorchAO.\n\nimport torch\nimport torch.nn as nn\nfrom torchao.quantization import (\n    IntxWeightOnlyConfig,\n    MappingType,\n    PerAxis,\n    quantize_,\n)\n\nif not torch.cuda.is_available():\n    raise RuntimeError(\"This example requires a CUDA GPU\")\n\nmodel = nn.Sequential(\n    nn.Linear(2048, 2048, bias=False),\n    nn.ReLU(),\n    nn.Linear(2048, 2048, bias=False),\n).eval().to(device=\"cuda\", dtype=torch.bfloat16)\n\nconfig = IntxWeightOnlyConfig(\n    weight_dtype=torch.int8,\n    granularity=PerAxis(0),\n    mapping_type=MappingType.ASYMMETRIC,\n)\nquantize_(model, config)\n\nx = torch.randn(1, 2048, device=\"cuda\", dtype=torch.bfloat16)\nwith torch.inference_mode():\n    output = model(x)\n\nprint(output.shape)\n\n123456789101112131415161718192021222324252627282930\n\nimport torchimport torch.nn as nnfrom torchao.quantization import (    IntxWeightOnlyConfig,    MappingType,    PerAxis,    quantize_,) if not torch.cuda.is_available():    raise RuntimeError(\"This example requires a CUDA GPU\") model = nn.Sequential(    nn.Linear(2048, 2048, bias=False),    nn.ReLU(),    nn.Linear(2048, 2048, bias=False),).eval().to(device=\"cuda\", dtype=torch.bfloat16) config = IntxWeightOnlyConfig(    weight_dtype=torch.int8,    granularity=PerAxis(0),    mapping_type=MappingType.ASYMMETRIC,)quantize_(model, config) x = torch.randn(1, 2048, device=\"cuda\", dtype=torch.bfloat16)with torch.inference_mode():    output = model(x) print(output.shape)\n```\n\n`PerAxis(0)` assigns separate quantization parameters to the rows of each linear weight matrix. `IntxWeightOnlyConfig` also supports per-group granularity. Its available packing formats and optimized kernels depend on the TorchAO version and target device.\n\nTorchAO also provides hardware-oriented presets such as `Int8WeightOnlyConfig` and `Int4WeightOnlyConfig`. These presets may use a different mapping from the explicit asymmetric example. You should use the configuration documented for the target hardware, then benchmark the complete model. A mathematically valid configuration is not necessarily the fastest one.\n\nThe `transformers` library can apply a TorchAO configuration while loading a model. For example:\n\n``` python\nLoad a causal model with a TorchAO quantization configuration.\n\nimport torch\nfrom torchao.quantization import Int4WeightOnlyConfig\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, TorchAoConfig\n\nmodel_id = \"Qwen/Qwen2.5-0.5B-Instruct\"\nquantization_config = TorchAoConfig(\n    quant_type=Int4WeightOnlyConfig(\n        group_size=128,\n        int4_packing_format=\"tile_packed_to_4d\",\n        int4_choose_qparams_algorithm=\"hqq\",\n    )\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_id,\n    dtype=torch.bfloat16,\n    device_map=\"auto\",\n    quantization_config=quantization_config,\n)\ntokenizer = AutoTokenizer.from_pretrained(model_id)\n\ninputs = tokenizer(\"Explain post-training quantization.\", return_tensors=\"pt\")\ninputs = inputs.to(model.device)\nwith torch.inference_mode():\n    generated = model.generate(**inputs, max_new_tokens=32)\n\nprint(tokenizer.decode(generated[0], skip_special_tokens=True))\n\n123456789101112131415161718192021222324252627\n\nimport torchfrom torchao.quantization import Int4WeightOnlyConfigfrom transformers import AutoModelForCausalLM, AutoTokenizer, TorchAoConfig model_id = \"Qwen/Qwen2.5-0.5B-Instruct\"quantization_config = TorchAoConfig(    quant_type=Int4WeightOnlyConfig(        group_size=128,        int4_packing_format=\"tile_packed_to_4d\",        int4_choose_qparams_algorithm=\"hqq\",    )) model = AutoModelForCausalLM.from_pretrained(    model_id,    dtype=torch.bfloat16,    device_map=\"auto\",    quantization_config=quantization_config,)tokenizer = AutoTokenizer.from_pretrained(model_id) inputs = tokenizer(\"Explain post-training quantization.\", return_tensors=\"pt\")inputs = inputs.to(model.device)with torch.inference_mode():    generated = model.generate(**inputs, max_new_tokens=32) print(tokenizer.decode(generated[0], skip_special_tokens=True))\n```\n\nThis is calibration-free weight-only PTQ. The explicit HQQ parameter choice and tiled packing avoid an optional MSLK kernel dependency in TorchAO 0.17 on the tested RTX 4090. Other TorchAO versions and devices may support different layouts. Static activation quantization is a different workflow: it prepares an exported model, runs representative inputs to collect activation ranges, and then converts the model.\n\n## GPTQ and AWQ\n\nRound-to-nearest quantization treats each weight locally. GPTQ and AWQ use calibration activations to reduce the effect of quantization error on model outputs.\n\nRounding each weight independently ignores how often and how strongly its input channel contributes to outputs. Calibration examples let a quantizer estimate this effect and reduce output error, though the result depends on how representative those examples are.\n\nGPTQ quantizes weights layer by layer using approximate second-order information. As weights are quantized, it adjusts the remaining weights to compensate for earlier error. The `transformers` library now uses GPT-QModel as its GPTQ backend. Install the required packages to use GPTQ for quantization:\n\n```\nInstall the GPTQ conversion dependencies.\n\npython -m pip install setuptools --upgrade wheel accelerate optimum transformers\npython -m pip install gptqmodel --no-build-isolation\n\n12\n\npython -m pip install setuptools --upgrade wheel accelerate optimum transformerspython -m pip install gptqmodel --no-build-isolation\n```\n\nThen you can quantize and save a small model. The `GPTQConfig` setting `dataset=\"c4\"` provides input samples for calibration. Setting `sym=False` requests the asymmetric form. Below is an example of quantizing a small model to INT4:\n\n``` python\nCalibrate and save an INT4 GPTQ checkpoint.\n\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig\n\nmodel_id = \"facebook/opt-125m\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nquantization_config = GPTQConfig(\n    bits=4,  # 2, 3, 4, or 8\n    group_size=128,\n    dataset=\"c4\",\n    tokenizer=tokenizer,\n    sym=False,\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_id,\n    device_map=\"auto\",\n    quantization_config=quantization_config,\n)\nmodel.save_pretrained(\"opt-125m-gptq\")\ntokenizer.save_pretrained(\"opt-125m-gptq\")\nprint(\"Saved the GPTQ checkpoint to opt-125m-gptq\")\n\n1234567891011121314151617181920\n\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig model_id = \"facebook/opt-125m\"tokenizer = AutoTokenizer.from_pretrained(model_id)quantization_config = GPTQConfig(    bits=4,  # 2, 3, 4, or 8    group_size=128,    dataset=\"c4\",    tokenizer=tokenizer,    sym=False,) model = AutoModelForCausalLM.from_pretrained(    model_id,    device_map=\"auto\",    quantization_config=quantization_config,)model.save_pretrained(\"opt-125m-gptq\")tokenizer.save_pretrained(\"opt-125m-gptq\")print(\"Saved the GPTQ checkpoint to opt-125m-gptq\")\n```\n\nTo load the saved GPTQ checkpoint in a fresh process and run one inference:\n\n``` python\nReload the saved GPTQ checkpoint and generate text.\n\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ncheckpoint = \"opt-125m-gptq\"\ntokenizer = AutoTokenizer.from_pretrained(checkpoint)\nmodel = AutoModelForCausalLM.from_pretrained(\n    checkpoint,\n    device_map=\"auto\",\n)\nmodel.eval()\n\ninputs = tokenizer(\"Weight quantization makes inference\", return_tensors=\"pt\")\ninputs = inputs.to(model.device)\n\nwith torch.inference_mode():\n    output = model.generate(\n        **inputs,\n        max_new_tokens=32,\n        pad_token_id=tokenizer.eos_token_id,\n    )\n\nprint(tokenizer.decode(output[0], skip_special_tokens=True))\n\n12345678910111213141516171819202122\n\nimport torchfrom transformers import AutoModelForCausalLM, AutoTokenizer checkpoint = \"opt-125m-gptq\"tokenizer = AutoTokenizer.from_pretrained(checkpoint)model = AutoModelForCausalLM.from_pretrained(    checkpoint,    device_map=\"auto\",)model.eval() inputs = tokenizer(\"Weight quantization makes inference\", return_tensors=\"pt\")inputs = inputs.to(model.device) with torch.inference_mode():    output = model.generate(        **inputs,        max_new_tokens=32,        pad_token_id=tokenizer.eos_token_id,    ) print(tokenizer.decode(output[0], skip_special_tokens=True))\n```\n\nAsymmetric GPTQ checkpoints have fewer compatible kernels than the default mapping. For example, the Marlin backend does not support asymmetric GPTQ. You should choose the quantization scheme and inference backend together.\n\nActivation-aware Weight Quantization (AWQ) is another quantization method. It identifies weights that are important under representative activations and searches for scaling factors that protect them before low-bit quantization. The `--model_path` argument points to the original, unquantized model in Hugging Face format. From a checkout of the reference `llm-awq` implementation with its dependencies installed, run the search first and then create the real INT4 checkpoint:\n\nInstall the reference implementation in a separate environment before running its commands:\n\n```\nInstall AutoAWQ and the Hugging Face Hub client.\n\ngit clone https://github.com/mit-han-lab/llm-awq.git\ncd llm-awq\npython -m pip install -e .\npython -m pip install huggingface_hub\n\n1234\n\ngit clone https://github.com/mit-han-lab/llm-awq.gitcd llm-awqpython -m pip install -e .python -m pip install huggingface_hub\n```\n\nThe reference project documents Python 3.10 for this environment. Use its [installation instructions](https://github.com/mit-han-lab/llm-awq#install) for the matching CUDA extensions. Download the model with\n\n```\nhf download TinyLlama/TinyLlama-1.1B-Chat-v1.0 --local-dir models/tinyllama\n\n1\n\nhf download TinyLlama/TinyLlama-1.1B-Chat-v1.0 --local-dir models/tinyllama\n```\n\nthen run the following commands:\n\n```\nSearch for AWQ scales and quantize the TinyLlama checkpoint.\n\n# Search for AWQ scaling parameters.\npython -m awq.entry \\\n  --model_path models/tinyllama \\\n  --w_bit 4 \\\n  --q_group_size 128 \\\n  --run_awq \\\n  --dump_awq awq_cache/tinyllama-w4-g128.pt\n\n# Create the quantized checkpoint.\npython -m awq.entry \\\n  --model_path models/tinyllama \\\n  --w_bit 4 \\\n  --q_group_size 128 \\\n  --load_awq awq_cache/tinyllama-w4-g128.pt \\\n  --q_backend real \\\n  --dump_quant quant_cache/tinyllama-w4-g128-awq-v2.pt\n\n12345678910111213141516\n\n# Search for AWQ scaling parameters.python -m awq.entry \\  --model_path models/tinyllama \\  --w_bit 4 \\  --q_group_size 128 \\  --run_awq \\  --dump_awq awq_cache/tinyllama-w4-g128.pt # Create the quantized checkpoint.python -m awq.entry \\  --model_path models/tinyllama \\  --w_bit 4 \\  --q_group_size 128 \\  --load_awq awq_cache/tinyllama-w4-g128.pt \\  --q_backend real \\  --dump_quant quant_cache/tinyllama-w4-g128-awq-v2.pt\n```\n\nThe `--run_awq` command obtains its calibration input internally. It uses 128 samples from the validation split of `mit-han-lab/pile-val-backup` and forms sequences of 512 tokens. This reference CLI does not expose an argument for selecting another text dataset. Omitting `--no_zero_point` keeps the implementation’s zero-point quantization.\n\nThe raw `.pt` file produced by `--dump_quant` is not a complete Hugging Face checkpoint. It contains the quantized weights but not the model configuration or tokenizer. The following script packages them together in a local directory:\n\n```\nPackage an AWQ weight file with its model configuration and tokenizer.\n\nfrom pathlib import Path\nimport shutil\n\nfrom transformers import AutoConfig, AutoTokenizer, AwqConfig\n\nsource_model = \"models/tinyllama\"\nquantized_weights = \"quant_cache/tinyllama-w4-g128-awq-v2.pt\"\noutput_dir = Path(\"models/tinyllama-awq\")\noutput_dir.mkdir(parents=True, exist_ok=True)\n\nawq_config = AwqConfig(\n    bits=4,\n    group_size=128,\n    zero_point=True,\n    backend=\"llm-awq\",\n    version=\"gemv\",\n)\n\nconfig = AutoConfig.from_pretrained(source_model)\nconfig.quantization_config = awq_config\nconfig.save_pretrained(output_dir)\n\ntokenizer = AutoTokenizer.from_pretrained(source_model)\ntokenizer.save_pretrained(output_dir)\n\nshutil.copyfile(\n    quantized_weights,\n    output_dir / \"pytorch_model.bin\",\n)\n\n1234567891011121314151617181920212223242526272829\n\nfrom pathlib import Pathimport shutil from transformers import AutoConfig, AutoTokenizer, AwqConfig source_model = \"models/tinyllama\"quantized_weights = \"quant_cache/tinyllama-w4-g128-awq-v2.pt\"output_dir = Path(\"models/tinyllama-awq\")output_dir.mkdir(parents=True, exist_ok=True) awq_config = AwqConfig(    bits=4,    group_size=128,    zero_point=True,    backend=\"llm-awq\",    version=\"gemv\",) config = AutoConfig.from_pretrained(source_model)config.quantization_config = awq_configconfig.save_pretrained(output_dir) tokenizer = AutoTokenizer.from_pretrained(source_model)tokenizer.save_pretrained(output_dir) shutil.copyfile(    quantized_weights,    output_dir / \"pytorch_model.bin\",)\n```\n\nThe `transformers` library can then identify the AWQ format from the checkpoint’s `quantization_config`, load the quantized weights, and run inference:\n\n``` python\nLoad the packaged AWQ checkpoint and generate text.\n\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_id = \"models/tinyllama-awq\"\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_id,\n    device_map=\"auto\",\n)\nmodel.eval()\n\ninputs = tokenizer(\"Explain weight quantization.\", return_tensors=\"pt\")\ninputs = inputs.to(model.device)\n\nwith torch.inference_mode():\n    output = model.generate(\n        **inputs,\n        max_new_tokens=64,\n        pad_token_id=tokenizer.eos_token_id,\n    )\n\nprint(tokenizer.decode(output[0], skip_special_tokens=True))\n\n12345678910111213141516171819202122\n\nimport torchfrom transformers import AutoModelForCausalLM, AutoTokenizer model_id = \"models/tinyllama-awq\"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(    model_id,    device_map=\"auto\",)model.eval() inputs = tokenizer(\"Explain weight quantization.\", return_tensors=\"pt\")inputs = inputs.to(model.device) with torch.inference_mode():    output = model.generate(        **inputs,        max_new_tokens=64,        pad_token_id=tokenizer.eos_token_id,    ) print(tokenizer.decode(output[0], skip_special_tokens=True))\n```\n\nThe model architecture, group size, zero-point convention, packed layout, and installed AWQ backend must be compatible.\n\nQuantizing a large model can take substantial time and memory. Before starting, check whether a compatible quantized checkpoint already exists. If you quantize it yourself, save the tokenizer and complete quantization configuration with the packed weights.\n\n## Quantizing a Model to GGUF\n\nGGUF is a container format used by the `llama.cpp` ecosystem. It stores model metadata, tokenizer information, tensors, and their types in one or more files. GGUF supports many quantization types; it is not itself one quantization algorithm.\n\nThe `llama.cpp` workflow has two stages. First, convert the original Hugging Face model to a high-precision GGUF file (such as bfloat16), then quantize that file. The following example uses the original, unquantized `TinyLlama/TinyLlama-1.1B-Chat-v1.0` checkpoint. From a built `llama.cpp` checkout:\n\nInstall the conversion script’s Python dependencies in a separate environment and build the binaries first. The conversion requirements may select a different PyTorch build from the one used for the CUDA examples earlier in this post. This CUDA build needs CMake, a C++ compiler, and the CUDA toolkit; omit `-DGGML_CUDA=ON` for a CPU build.\n\n```\nClone and build llama.cpp with GGUF conversion support.\n\ngit clone https://github.com/ggml-org/llama.cpp.git\ncd llama.cpp\npython -m pip install -r requirements/requirements-convert_hf_to_gguf.txt\npython -m pip install huggingface_hub\ncmake -B build -DGGML_CUDA=ON\ncmake --build build --config Release\nhf download TinyLlama/TinyLlama-1.1B-Chat-v1.0 --local-dir models/tinyllama\n\n1234567\n\ngit clone https://github.com/ggml-org/llama.cpp.gitcd llama.cpppython -m pip install -r requirements/requirements-convert_hf_to_gguf.txtpython -m pip install huggingface_hubcmake -B build -DGGML_CUDA=ONcmake --build build --config Releasehf download TinyLlama/TinyLlama-1.1B-Chat-v1.0 --local-dir models/tinyllama\nConvert TinyLlama to GGUF and quantize it to Q4_K_M.\n\npython3 convert_hf_to_gguf.py models/tinyllama \\\n  --outfile tinyllama-bf16.gguf \\\n  --outtype bf16\n\n./build/bin/llama-quantize tinyllama-bf16.gguf tinyllama-Q4_K_M.gguf Q4_K_M\n\n12345\n\npython3 convert_hf_to_gguf.py models/tinyllama \\  --outfile tinyllama-bf16.gguf \\  --outtype bf16 ./build/bin/llama-quantize tinyllama-bf16.gguf tinyllama-Q4_K_M.gguf Q4_K_M\n```\n\n`Q4_K_M` is a mixed GGUF quantization type that uses different encodings for different tensors. Its file size includes scales, metadata, and tensors stored at other precisions, so it is not exactly four bits per model parameter. Its name does not mean every value follows the affine formula shown earlier.\n\nYou can use `llama-cli` to load the quantized GGUF file and generate text directly from its packed weights:\n\n```\nRun the quantized GGUF model with llama.cpp.\n\n./build/bin/llama-cli \\\n  -m tinyllama-Q4_K_M.gguf \\\n  -p \"Explain weight-only quantization in one paragraph.\" \\\n  -n 128 --single-turn\n\n1234\n\n./build/bin/llama-cli \\  -m tinyllama-Q4_K_M.gguf \\  -p \"Explain weight-only quantization in one paragraph.\" \\  -n 128 --single-turn\n```\n\nThis confirms that `llama.cpp` can generate from the packed file. TinyLlama may give an inaccurate explanation, so use separate quality checks for your task.\n\nSome GGUF models are available on the Hugging Face Hub and can be run directly:\n\n```\nRun a GGUF model directly from the Hugging Face Hub.\n\n./build/bin/llama-cli -hf Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M \\\n  -p \"What is a quantized model?\" -n 64 --single-turn\n\n12\n\n./build/bin/llama-cli -hf Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M \\  -p \"What is a quantized model?\" -n 64 --single-turn\n```\n\nIf you prefer to use Python code, `transformers` also accepts a GGUF filename for supported architectures:\n\n```\nInstall the Hugging Face library with GGUF loading support.\n\npython -m pip install --upgrade transformers gguf\n\n1\n\npython -m pip install --upgrade transformers gguf\n```\n\nUse a recent `transformers` release for the Qwen3.5 example below. Loading may dequantize tensors and require much more memory than the GGUF file size suggests.\n\n```\nLoad a GGUF checkpoint through the Hugging Face library.\n\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_id = \"unsloth/Qwen3.5-4B-GGUF\"\nfilename = \"Qwen3.5-4B-Q4_K_M.gguf\"\n\nmodel = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)\ntokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)\nprint(type(model).__name__, type(tokenizer).__name__)\n\n12345678\n\nfrom transformers import AutoModelForCausalLM, AutoTokenizer model_id = \"unsloth/Qwen3.5-4B-GGUF\"filename = \"Qwen3.5-4B-Q4_K_M.gguf\" model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)print(type(model).__name__, type(tokenizer).__name__)\n```\n\nBut you should not assume that `transformers` preserves compressed inference. It keeps GGUF blocks packed only for a narrow set of model architectures, devices, and quantization types; other combinations are dequantized while loading. You can use `llama.cpp` when broad GGUF compatibility and direct execution of packed weights are required. You should quantize from the highest-quality source available. Requantizing an already quantized model compounds approximation error, and `llama-quantize` warns that it can severely reduce quality.\n\n## Evaluating Weight-Only Quantization\n\nEvaluate the quantized model on representative inputs. If quality degrades beyond your application’s tolerance, try a different calibration set, quantizer, or bit width.\n\nSome quantizers use calibration examples to reduce output error between the original and quantized model. That proxy does not replace application quality metrics. Useful checks include:\n\n- Perplexity on representative text\n- Accuracy or task-specific scores\n- Compare greedy generations side by side\n- Compare sampled generations across several seeds, prompt lengths, and temperatures\n\nAfter creating `opt-125m-gptq` with the earlier command, run this script on one CUDA GPU. It loads and measures the FP16 baseline and the GPTQ checkpoint *one at a time* so their peak allocated memory measurements are comparable. The timing includes `generate()` but excludes model loading and tokenization:\n\n``` python\nCompare memory use and generation time across model checkpoints.\n\nimport gc\nimport time\n\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nif not torch.cuda.is_available():\n    raise RuntimeError(\"This comparison needs a CUDA GPU and a GPTQ CUDA kernel\")\n\ntokenizer = AutoTokenizer.from_pretrained(\"facebook/opt-125m\")\ninputs = tokenizer(\"Weight quantization makes inference\", return_tensors=\"pt\").to(\"cuda\")\n\ndef measure_checkpoint(path):\n    torch.cuda.empty_cache()\n    torch.cuda.reset_peak_memory_stats()\n    model = AutoModelForCausalLM.from_pretrained(\n        path, device_map=\"cuda:0\", dtype=torch.float16,\n    ).eval()\n\n    def generate(current_model):\n        with torch.inference_mode():\n            return current_model.generate(\n                **inputs, max_new_tokens=32, do_sample=False,\n                pad_token_id=tokenizer.eos_token_id,\n            )\n\n    generate(model)  # warm up\n    torch.cuda.synchronize()\n    times = []\n    for _ in range(3):\n        start = time.perf_counter()\n        ids = generate(model)\n        torch.cuda.synchronize()\n        times.append(time.perf_counter() - start)\n\n    result = {\n        \"median_seconds\": sorted(times)[1],\n        \"peak_allocated_gib\": torch.cuda.max_memory_allocated() / 2**30,\n        \"output\": tokenizer.decode(ids[0], skip_special_tokens=True),\n    }\n    del model\n    gc.collect()\n    torch.cuda.empty_cache()\n    return result\n\nfor checkpoint in (\"facebook/opt-125m\", \"opt-125m-gptq\"):\n    print(checkpoint, measure_checkpoint(checkpoint))\n\n1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950\n\nimport gcimport time import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer  if not torch.cuda.is_available():    raise RuntimeError(\"This comparison needs a CUDA GPU and a GPTQ CUDA kernel\") tokenizer = AutoTokenizer.from_pretrained(\"facebook/opt-125m\")inputs = tokenizer(\"Weight quantization makes inference\", return_tensors=\"pt\").to(\"cuda\")  def measure_checkpoint(path):    torch.cuda.empty_cache()    torch.cuda.reset_peak_memory_stats()    model = AutoModelForCausalLM.from_pretrained(        path, device_map=\"cuda:0\", dtype=torch.float16,    ).eval()     def generate(current_model):        with torch.inference_mode():            return current_model.generate(                **inputs, max_new_tokens=32, do_sample=False,                pad_token_id=tokenizer.eos_token_id,            )     generate(model)  # warm up    torch.cuda.synchronize()    times = []    for _ in range(3):        start = time.perf_counter()        ids = generate(model)        torch.cuda.synchronize()        times.append(time.perf_counter() - start)     result = {        \"median_seconds\": sorted(times)[1],        \"peak_allocated_gib\": torch.cuda.max_memory_allocated() / 2**30,        \"output\": tokenizer.decode(ids[0], skip_special_tokens=True),    }    del model    gc.collect()    torch.cuda.empty_cache()    return result  for checkpoint in (\"facebook/opt-125m\", \"opt-125m-gptq\"):    print(checkpoint, measure_checkpoint(checkpoint))\n```\n\nThis is a small diagnostic benchmark. For deployment decisions, use the same prompts, output limits, warmup, and concurrency as the real workload. You should measure perplexity or task accuracy as well as sample generations; one matching greedy answer does not establish quality parity. Peak PyTorch allocation may omit memory held by other libraries, so check device-level memory too.\n\nA quantized model may require specialized kernels or a compatible inference engine. You may need specialized hardware for the best performance. You can compare the following measurements on the target platform to decide if quantization is worth the effort:\n\n- Checkpoint size and model-load memory\n- Peak memory during prefill and decode\n- Time to first token\n- Throughput at realistic batch sizes and concurrency\n\n## Further Reading\n\nBelow are some resources you may find useful:\n\n- PyTorch TorchAO documentation. [Quantized Inference](https://docs.pytorch.org/ao/stable/workflows/inference.html) . This guide lists current weight-only and dynamic inference configurations.\n- PyTorch TorchAO documentation. [quantize_](https://docs.pytorch.org/ao/stable/api_reference/generated/torchao.quantization.quantize_.html) . This page documents the in-place post-training quantization API.\n- Elias Frantar et al. (2022). [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://arxiv.org/abs/2210.17323) . This paper introduces the GPTQ algorithm.\n- Hugging Face Transformers documentation. [GPTQ](https://huggingface.co/docs/transformers/main/en/quantization/gptq) . This guide covers quantization and checkpoint loading with GPT-QModel.\n- Ji Lin et al. (2023). [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://arxiv.org/abs/2306.00978) . This paper introduces AWQ.\n- [AWQ reference implementation](https://github.com/mit-han-lab/llm-awq) . By MIT Han Lab. This repository includes commands for searching AWQ parameters and producing real quantized weights.\n- [AWQ conversion example](https://github.com/mit-han-lab/llm-awq/blob/main/examples/convert_to_hf.py) . In the reference repository. This shows how to package a raw quantized state dictionary with a Transformers configuration and tokenizer.\n- [LLM Compressor](https://github.com/vllm-project/llm-compressor) . By the vLLM project. This toolkit provides GPTQ, AWQ, and other post-training compression recipes.\n- [llama.cpp quantization](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) . In the llama.cpp repository. This guide documents conversion, quantization types, importance matrices, and execution.\n- Hugging Face Transformers documentation. [GGUF](https://huggingface.co/docs/transformers/main/en/quantization/gguf) . This page explains the supported GGUF loading paths and fallback dequantization.\n- [NVIDIA A100 Tensor Core GPU](https://www.nvidia.com/en-us/data-center/a100/) . In the NVIDIA documentation. Its specifications compare peak throughput across FP16, INT8, and other numerical formats.\n- [AMD Instinct MI300X Platform](https://www.amd.com/en/products/accelerators/instinct/mi300/platform.html) . In the AMD documentation. Its specifications provide another example of higher peak INT8 matrix throughput than FP16 throughput.\n\n## Summary\n\nIn this post, you learned how asymmetric affine quantization maps floating-point weights to integer codes using a scale and zero point. Per-tensor, per-channel, and per-group parameters trade metadata and kernel complexity for accuracy. Weight-only post-training quantization derives these parameters after training while keeping activations at higher precision.\n\nTorchAO provides PyTorch-native PTQ, while GPTQ and AWQ use calibration data to preserve model behavior at low bit widths. GPT-QModel, LLM Compressor, and the AWQ reference implementation provide workflows outside TorchAO. GGUF packages quantized tensors for runtimes such as `llama.cpp`.\n\nQuantization is useful only when the checkpoint format, numerical scheme, packed layout, and inference kernel work together. Evaluate quality, memory, latency, throughput, and compatibility on the actual deployment target.", "url": "https://wpnews.pro/news/weight-only-quantization", "canonical_source": "https://machinelearningmastery.com/weight-only-quantization/", "published_at": "2026-11-14 16:04:34+00:00", "updated_at": "2026-11-14 18:50:35.406888+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-infrastructure", "ai-tools"], "entities": ["Machine Learning Mastery", "PyTorch", "TorchAO", "GPTQ", "AWQ", "GGUF", "llama.cpp", "Transformers"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/weight-only-quantization", "markdown": "https://wpnews.pro/news/weight-only-quantization.md", "text": "https://wpnews.pro/news/weight-only-quantization.txt", "jsonld": "https://wpnews.pro/news/weight-only-quantization.jsonld"}}