{"slug": "floating-point-precision-and-its-effects-to-model-inference", "title": "Floating-Point Precision and Its Effects to Model Inference", "summary": "A tutorial on floating-point precision for model inference advises that the useful question is not which format has the fewest bits but which format meets the quality requirement and is implemented efficiently on the deployment hardware. The post explains that float16 (FP16) has more significand bits than bfloat16 (BF16) and so distinguishes nearby values more precisely, while BF16 has the same number of exponent bits as float32 (FP32) and supports a much larger dynamic range, often avoiding overflow on transformer activations that FP16 would encounter. It walks through loading a model at a different precision, handling sensitive operations with mixed precision, and measuring the performance and quality effects of a precision change, noting that a matrix multiplication can read FP16 operands, accumulate partial sums in FP32, and write an FP16 result.", "body_md": "A model is a collection of numbers. The format used to store and compute with those numbers affects how much memory the model occupies, how quickly the hardware can move the numbers, and which arithmetic units can process them. Changing precision can therefore make inference faster without changing the model architecture.\n\nLower precision is not automatically better. A format may save memory but run slowly when the target hardware lacks a suitable kernel. It may represent large values but lose small differences, or preserve fine differences near zero but overflow on an activation outlier. The useful question is not which format has the fewest bits. It is which format meets the quality requirement and is implemented efficiently on the deployment hardware.\n\nIn this post, you will learn about:\n\n- How floating-point formats trade range, precision, and memory use\n- How to load a model in a suitable precision and handle sensitive operations\n- How to measure the performance and quality effects of a precision change\n\nLet’s get started.\n\n## Overview\n\nThis post is divided into five parts; they are:\n\n- How Floating-Point Numbers Work\n- Common Precision Formats\n- Loading a Model at a Different Precision\n- Mixed Precision and Sensitive Operations\n- Measuring the Result\n\n## How Floating-Point Numbers Work\n\nA floating-point number contains a sign, an exponent, and a significand (often called the mantissa). The exponent determines the range of magnitudes that can be represented. The significand determines how precisely nearby values can be distinguished. Reducing either field creates a trade-off.\n\nThe figure above shows a normal finite number. The leading $1$ in its significand is implicit, so 23 stored fraction bits provide 24 bits of precision. The all-zero and all-one exponent fields encode subnormal numbers or zero, and infinities or NaNs, respectively; they do not use the normal-number formula in the figure.\n\nFour effects are important during inference:\n\n- *Rounding:* A value between two representable numbers is replaced by one of them.\n- *Underflow:* A very small value becomes zero or loses most of its significant bits.\n- *Overflow:* A value is too large for the format and becomes infinity.\n- *Accumulation error:* Many rounded products are added, allowing small errors to accumulate.\n\nPyTorch exposes these limits through `torch.finfo()`:\n\n``` python\nPrint the numerical limits of common PyTorch floating-point types.\n\nimport torch\n\nfor dtype in (torch.float32, torch.float16, torch.bfloat16):\n    info = torch.finfo(dtype)\n    print(\n        dtype,\n        \"smallest_normal=\", info.tiny,\n        \"largest=\", info.max,\n        \"resolution=\", info.eps,\n    )\n\n12345678910\n\nimport torch for dtype in (torch.float32, torch.float16, torch.bfloat16):    info = torch.finfo(dtype)    print(        dtype,        \"smallest_normal=\", info.tiny,        \"largest=\", info.max,        \"resolution=\", info.eps,    )\n```\n\nThe `float16` (FP16) format has more significand bits than `bfloat16` (BF16), so it distinguishes nearby values more precisely. The `bfloat16` format has the same number of exponent bits as `float32` (FP32), so it supports a much larger dynamic range than `float16`. For transformer activations, BF16’s wider range often avoids overflow that FP16 would encounter. The better choice still depends on the model, kernels, and hardware.\n\nThe data type (`dtype`) of an input tensor does not completely describe the arithmetic. A matrix multiplication can read FP16 operands, accumulate partial sums in FP32, and write an FP16 result. A higher precision intermediate result is often used to avoid overflow and underflow. Hardware and library settings determine the exact path. When documenting a benchmark, record operand, accumulation, and output precision when possible.\n\n## Common Precision Formats\n\nFP32 is the 32-bit single-precision format and a useful numerical reference. It uses twice the parameter memory of a 16-bit format. FP64 is useful for some scientific calculations, but its storage and arithmetic cost seldom help LLM inference. Support for wider types varies by software and hardware; there is no universal drop-in `FP128` inference path.\n\nFP16 uses 16 bits with a five-bit exponent. It reduces model storage and memory traffic, and modern accelerators have fast FP16 matrix multiplication. Its limited range makes overflow possible when activations contain large values.\n\nBF16 also uses 16 bits, but it allocates eight bits to the exponent and fewer bits to the significand. Its range is close to FP32, while its precision is lower. Recent data-center GPUs and some CPUs have fast BF16 arithmetic. PyTorch also supports BF16 operations and autocast on CPUs, although unsupported operations may use a different dtype or run slowly. You should check the actual device and kernels before choosing it.\n\nFP8 formats use eight bits. FP8 reduces model memory use when 16 bits per value is too costly. There is more than one FP8 layout because eight bits cannot provide both a wide range and high precision. E4M3 has more significand precision, while E5M2 has more range. Inference engines may choose different formats for weights, activations, and gradients, and typically maintain scaling information outside the eight-bit value.\n\nFP4 formats reduce storage further. Four bits are too few for a general drop-in replacement for FP16, so practical systems use carefully selected encodings, block scales, calibration, or training recipes. MXFP4 and NVFP4 specify different scaling and layout rules; accelerator support depends on the exact device generation and software stack. You should verify a specific format against the runtime’s compatibility table.\n\nThe parameter memory of a dense model is approximately:\n\n```\nparameter_memory = parameter_count * bits_per_parameter / 8\n\n1\n\nparameter_memory = parameter_count * bits_per_parameter / 8\n```\n\nA 7B parameter model needs about 28 GB for FP32 weights, 14 GB for FP16 or BF16 weights, and 7 GB for eight-bit weights before metadata and runtime buffers are counted. This estimate excludes the KV cache, activations, allocator overhead, and temporary workspace. Eight-bit *scaled quantization* also stores scales and possibly zero points; it is different from simply casting a tensor to another floating-point dtype.\n\n## Loading a Model at a Different Precision\n\nPyTorch models allow you to convert floating-point parameters and buffers with `to()`. For example:\n\n``` python\nConvert a small PyTorch model to a different parameter dtype.\n\nimport torch\n\nmodel = torch.nn.Linear(8, 8).to(dtype=torch.float16)\nprint(model.weight.dtype)\n\n1234\n\nimport torch model = torch.nn.Linear(8, 8).to(dtype=torch.float16)print(model.weight.dtype)\n```\n\nThe code converts a small linear layer and prints `torch.float16`. For a pretrained model, select a dtype supported by the target device and load it directly in that dtype as shown next.\n\nHugging Face models can be loaded directly into a floating-point dtype using the `transformers` library:\n\n```\nLoad a language model in a selected floating-point dtype and generate text.\n\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_name = \"sshleifer/tiny-gpt2\"\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\ndtype = torch.float16 if device == \"cuda\" else torch.float32\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_name,\n    dtype=dtype,\n).to(device)\nmodel.eval()\n\ninputs = tokenizer(\"Efficient inference is\", return_tensors=\"pt\").to(device)\nwith torch.no_grad():\n    output = model.generate(**inputs, max_new_tokens=20)\n\nprint(tokenizer.decode(output[0], skip_special_tokens=True))\n\n1234567891011121314151617181920\n\nimport torchfrom transformers import AutoModelForCausalLM, AutoTokenizer  model_name = \"sshleifer/tiny-gpt2\"device = \"cuda\" if torch.cuda.is_available() else \"cpu\"dtype = torch.float16 if device == \"cuda\" else torch.float32 tokenizer = AutoTokenizer.from_pretrained(model_name)model = AutoModelForCausalLM.from_pretrained(    model_name,    dtype=dtype,).to(device)model.eval() inputs = tokenizer(\"Efficient inference is\", return_tensors=\"pt\").to(device)with torch.no_grad():    output = model.generate(**inputs, max_new_tokens=20) print(tokenizer.decode(output[0], skip_special_tokens=True))\n```\n\nLoading directly into the desired dtype avoids temporarily holding a full FP32 model and then converting it. This matters for large checkpoints. Some checkpoints already store tensors in a lower precision, while others are converted as they are loaded. Do not assume every device supports every dtype. Test a representative forward pass and inspect the profiler. An unsupported operation may fail, fall back to a slower implementation, or insert casts that erase the expected speedup.\n\nAlso note that FP16 has a narrower dynamic range than FP32, so it is possible to overflow or underflow when an FP32 model is converted to FP16.\n\n## Mixed Precision and Sensitive Operations\n\nA model does not have to use one dtype everywhere. Matrix multiplications may use FP16 or BF16 while normalization, reduction, logits, or selected layers remain in FP32. This is called mixed precision.\n\nMixed precision can reduce numerical error while keeping expensive matrix operations in a faster format. Reductions and softmax need particular care: subtracting the maximum logit before exponentiation, and accumulating in a wider dtype, can prevent overflow and reduce error. The exact internal dtype is determined by the kernel and is not always the dtype of the stored weights.\n\nFor a model you control, cast selected operations explicitly while keeping the boundary dtype consistent. This self-contained example computes normalization in FP32 and returns the input dtype:\n\n``` python\nCompute normalization in FP32 while returning the original dtype.\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nclass FP32LayerNorm(nn.Module):\n    def __init__(self, width):\n        super().__init__()\n        self.weight = nn.Parameter(torch.ones(width, dtype=torch.float32))\n        self.bias = nn.Parameter(torch.zeros(width, dtype=torch.float32))\n\n    def forward(self, x):\n        y = F.layer_norm(x.float(), (x.size(-1),), self.weight, self.bias)\n        return y.to(x.dtype)\n\nnorm = FP32LayerNorm(8)\nx = torch.randn(2, 8, dtype=torch.float16)\nprint(norm(x).dtype)\n\n1234567891011121314151617\n\nimport torchimport torch.nn as nnimport torch.nn.functional as F class FP32LayerNorm(nn.Module):    def __init__(self, width):        super().__init__()        self.weight = nn.Parameter(torch.ones(width, dtype=torch.float32))        self.bias = nn.Parameter(torch.zeros(width, dtype=torch.float32))     def forward(self, x):        y = F.layer_norm(x.float(), (x.size(-1),), self.weight, self.bias)        return y.to(x.dtype) norm = FP32LayerNorm(8)x = torch.randn(2, 8, dtype=torch.float16)print(norm(x).dtype)\n```\n\nDo not indiscriminately change modules in a pretrained model: a FP16 linear layer may reject an FP32 activation, and a later whole-model `to()` call would convert the selected norm back to FP16. Extra casts can also cost time. You should use the model’s supported mixed-precision path and validate outputs on representative inputs.\n\n## Measuring the Result\n\nPrecision changes should be evaluated along three axes:\n\n- *Memory:* Parameter storage, peak allocated memory, and KV cache capacity.\n- *Performance:* Prefill latency, decode latency, and throughput at realistic batch sizes and sequence lengths.\n- *Quality:* Perplexity, task metrics, and generation comparisons on representative prompts.\n\nYou can refer to the benchmarking rules and code examples from [the previous post](https://machinelearningmastery.com/measuring-performance-of-transformer-inference/): warm up the model, synchronize asynchronous devices, separate prefill from decode, and report the hardware and software configuration. Compare output logits in addition to decoded text when debugging. Greedy outputs can remain identical even when many logits have changed, or diverge after one close token decision.\n\nA precision format reduces memory only in proportion to the tensors converted. If weights are half of total runtime memory because the KV cache is large, halving weight storage cannot halve total memory. Likewise, lower-precision weights help decode most when weight movement is the bottleneck. Large-batch prefill may already be compute-bound.\n\n## Further Reading\n\nBelow are some resources you may find useful:\n\n- PyTorch documentation. [Floating point types](https://pytorch.org/docs/stable/tensor_attributes.html#torch.dtype) . This page lists the floating-point dtypes exposed by PyTorch.\n- PyTorch documentation. [Mixed precision numerical accuracy](https://docs.pytorch.org/docs/stable/notes/numerical_accuracy.html#reduced-precision-reduction-for-fp16-and-bf16-gemms) . This explains reduced-precision accumulation and numerical behavior in matrix multiplication.\n- PyTorch blog. [BF16 on Intel Xeon Processors](https://pytorch.org/blog/empowering-pytorch-on-intel-xeon-scalable-processors-with-bfloat16/) . This documents CPU BF16 operations and autocast support.\n- Paulius Micikevicius et al. (2022). [FP8 Formats for Deep Learning](https://arxiv.org/abs/2209.05433) . This paper describes the E4M3 and E5M2 FP8 formats and their use in deep learning.\n- Open Compute Project. [OCP Microscaling Formats](https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf) . This specification describes block-scaled low-precision formats, including microscaling formats used with very small element types.\n\n## Summary\n\nIn this post, you learned how floating-point formats trade dynamic range, precision, memory, and hardware efficiency. FP16 and BF16 both use 16 bits but behave differently because their exponent and significand fields differ. You also learned that computation may use a different accumulation dtype from storage and that selected operations may need higher precision.", "url": "https://wpnews.pro/news/floating-point-precision-and-its-effects-to-model-inference", "canonical_source": "https://machinelearningmastery.com/floating-point-precision/", "published_at": "2026-11-07 13:29:33+00:00", "updated_at": "2026-11-09 01:57:47.823557+00:00", "lang": "en", "topics": ["machine-learning", "ai-infrastructure", "large-language-models", "ai-tools"], "entities": ["PyTorch", "float16", "bfloat16", "float32", "float64"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/floating-point-precision-and-its-effects-to-model-inference", "markdown": "https://wpnews.pro/news/floating-point-precision-and-its-effects-to-model-inference.md", "text": "https://wpnews.pro/news/floating-point-precision-and-its-effects-to-model-inference.txt", "jsonld": "https://wpnews.pro/news/floating-point-precision-and-its-effects-to-model-inference.jsonld"}}