Floating-Point Precision and Its Effects to Model Inference

A tutorial on floating-point precision for model inference advises that the useful question is not which format has the fewest bits but which format meets the quality requirement and is implemented efficiently on the deployment hardware. The post explains that float16 (FP16) has more significand bits than bfloat16 (BF16) and so distinguishes nearby values more precisely, while BF16 has the same number of exponent bits as float32 (FP32) and supports a much larger dynamic range, often avoiding overflow on transformer activations that FP16 would encounter. It walks through loading a model at a different precision, handling sensitive operations with mixed precision, and measuring the performance and quality effects of a precision change, noting that a matrix multiplication can read FP16 operands, accumulate partial sums in FP32, and write an FP16 result.

A model is a collection of numbers. The format used to store and compute with those numbers affects how much memory the model occupies, how quickly the hardware can move the numbers, and which arithmetic units can process them. Changing precision can therefore make inference faster without changing the model architecture. Lower precision is not automatically better. A format may save memory but run slowly when the target hardware lacks a suitable kernel. It may represent large values but lose small differences, or preserve fine differences near zero but overflow on an activation outlier. The useful question is not which format has the fewest bits. It is which format meets the quality requirement and is implemented efficiently on the deployment hardware. In this post, you will learn about: - How floating-point formats trade range, precision, and memory use - How to load a model in a suitable precision and handle sensitive operations - How to measure the performance and quality effects of a precision change Let’s get started. Overview This post is divided into five parts; they are: - How Floating-Point Numbers Work - Common Precision Formats - Loading a Model at a Different Precision - Mixed Precision and Sensitive Operations - Measuring the Result How Floating-Point Numbers Work A floating-point number contains a sign, an exponent, and a significand often called the mantissa . The exponent determines the range of magnitudes that can be represented. The significand determines how precisely nearby values can be distinguished. Reducing either field creates a trade-off. The figure above shows a normal finite number. The leading $1$ in its significand is implicit, so 23 stored fraction bits provide 24 bits of precision. The all-zero and all-one exponent fields encode subnormal numbers or zero, and infinities or NaNs, respectively; they do not use the normal-number formula in the figure. Four effects are important during inference: - Rounding: A value between two representable numbers is replaced by one of them. - Underflow: A very small value becomes zero or loses most of its significant bits. - Overflow: A value is too large for the format and becomes infinity. - Accumulation error: Many rounded products are added, allowing small errors to accumulate. PyTorch exposes these limits through torch.finfo : python Print the numerical limits of common PyTorch floating-point types. import torch for dtype in torch.float32, torch.float16, torch.bfloat16 : info = torch.finfo dtype print dtype, "smallest normal=", info.tiny, "largest=", info.max, "resolution=", info.eps, 12345678910 import torch for dtype in torch.float32, torch.float16, torch.bfloat16 : info = torch.finfo dtype print dtype, "smallest normal=", info.tiny, "largest=", info.max, "resolution=", info.eps, The float16 FP16 format has more significand bits than bfloat16 BF16 , so it distinguishes nearby values more precisely. The bfloat16 format has the same number of exponent bits as float32 FP32 , so it supports a much larger dynamic range than float16 . For transformer activations, BF16’s wider range often avoids overflow that FP16 would encounter. The better choice still depends on the model, kernels, and hardware. The data type dtype of an input tensor does not completely describe the arithmetic. A matrix multiplication can read FP16 operands, accumulate partial sums in FP32, and write an FP16 result. A higher precision intermediate result is often used to avoid overflow and underflow. Hardware and library settings determine the exact path. When documenting a benchmark, record operand, accumulation, and output precision when possible. Common Precision Formats FP32 is the 32-bit single-precision format and a useful numerical reference. It uses twice the parameter memory of a 16-bit format. FP64 is useful for some scientific calculations, but its storage and arithmetic cost seldom help LLM inference. Support for wider types varies by software and hardware; there is no universal drop-in FP128 inference path. FP16 uses 16 bits with a five-bit exponent. It reduces model storage and memory traffic, and modern accelerators have fast FP16 matrix multiplication. Its limited range makes overflow possible when activations contain large values. BF16 also uses 16 bits, but it allocates eight bits to the exponent and fewer bits to the significand. Its range is close to FP32, while its precision is lower. Recent data-center GPUs and some CPUs have fast BF16 arithmetic. PyTorch also supports BF16 operations and autocast on CPUs, although unsupported operations may use a different dtype or run slowly. You should check the actual device and kernels before choosing it. FP8 formats use eight bits. FP8 reduces model memory use when 16 bits per value is too costly. There is more than one FP8 layout because eight bits cannot provide both a wide range and high precision. E4M3 has more significand precision, while E5M2 has more range. Inference engines may choose different formats for weights, activations, and gradients, and typically maintain scaling information outside the eight-bit value. FP4 formats reduce storage further. Four bits are too few for a general drop-in replacement for FP16, so practical systems use carefully selected encodings, block scales, calibration, or training recipes. MXFP4 and NVFP4 specify different scaling and layout rules; accelerator support depends on the exact device generation and software stack. You should verify a specific format against the runtime’s compatibility table. The parameter memory of a dense model is approximately: parameter memory = parameter count bits per parameter / 8 1 parameter memory = parameter count bits per parameter / 8 A 7B parameter model needs about 28 GB for FP32 weights, 14 GB for FP16 or BF16 weights, and 7 GB for eight-bit weights before metadata and runtime buffers are counted. This estimate excludes the KV cache, activations, allocator overhead, and temporary workspace. Eight-bit scaled quantization also stores scales and possibly zero points; it is different from simply casting a tensor to another floating-point dtype. Loading a Model at a Different Precision PyTorch models allow you to convert floating-point parameters and buffers with to . For example: python Convert a small PyTorch model to a different parameter dtype. import torch model = torch.nn.Linear 8, 8 .to dtype=torch.float16 print model.weight.dtype 1234 import torch model = torch.nn.Linear 8, 8 .to dtype=torch.float16 print model.weight.dtype The code converts a small linear layer and prints torch.float16 . For a pretrained model, select a dtype supported by the target device and load it directly in that dtype as shown next. Hugging Face models can be loaded directly into a floating-point dtype using the transformers library: Load a language model in a selected floating-point dtype and generate text. import torch from transformers import AutoModelForCausalLM, AutoTokenizer model name = "sshleifer/tiny-gpt2" device = "cuda" if torch.cuda.is available else "cpu" dtype = torch.float16 if device == "cuda" else torch.float32 tokenizer = AutoTokenizer.from pretrained model name model = AutoModelForCausalLM.from pretrained model name, dtype=dtype, .to device model.eval inputs = tokenizer "Efficient inference is", return tensors="pt" .to device with torch.no grad : output = model.generate inputs, max new tokens=20 print tokenizer.decode output 0 , skip special tokens=True 1234567891011121314151617181920 import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer model name = "sshleifer/tiny-gpt2"device = "cuda" if torch.cuda.is available else "cpu"dtype = torch.float16 if device == "cuda" else torch.float32 tokenizer = AutoTokenizer.from pretrained model name model = AutoModelForCausalLM.from pretrained model name, dtype=dtype, .to device model.eval inputs = tokenizer "Efficient inference is", return tensors="pt" .to device with torch.no grad : output = model.generate inputs, max new tokens=20 print tokenizer.decode output 0 , skip special tokens=True Loading directly into the desired dtype avoids temporarily holding a full FP32 model and then converting it. This matters for large checkpoints. Some checkpoints already store tensors in a lower precision, while others are converted as they are loaded. Do not assume every device supports every dtype. Test a representative forward pass and inspect the profiler. An unsupported operation may fail, fall back to a slower implementation, or insert casts that erase the expected speedup. Also note that FP16 has a narrower dynamic range than FP32, so it is possible to overflow or underflow when an FP32 model is converted to FP16. Mixed Precision and Sensitive Operations A model does not have to use one dtype everywhere. Matrix multiplications may use FP16 or BF16 while normalization, reduction, logits, or selected layers remain in FP32. This is called mixed precision. Mixed precision can reduce numerical error while keeping expensive matrix operations in a faster format. Reductions and softmax need particular care: subtracting the maximum logit before exponentiation, and accumulating in a wider dtype, can prevent overflow and reduce error. The exact internal dtype is determined by the kernel and is not always the dtype of the stored weights. For a model you control, cast selected operations explicitly while keeping the boundary dtype consistent. This self-contained example computes normalization in FP32 and returns the input dtype: python Compute normalization in FP32 while returning the original dtype. import torch import torch.nn as nn import torch.nn.functional as F class FP32LayerNorm nn.Module : def init self, width : super . init self.weight = nn.Parameter torch.ones width, dtype=torch.float32 self.bias = nn.Parameter torch.zeros width, dtype=torch.float32 def forward self, x : y = F.layer norm x.float , x.size -1 , , self.weight, self.bias return y.to x.dtype norm = FP32LayerNorm 8 x = torch.randn 2, 8, dtype=torch.float16 print norm x .dtype 1234567891011121314151617 import torchimport torch.nn as nnimport torch.nn.functional as F class FP32LayerNorm nn.Module : def init self, width : super . init self.weight = nn.Parameter torch.ones width, dtype=torch.float32 self.bias = nn.Parameter torch.zeros width, dtype=torch.float32 def forward self, x : y = F.layer norm x.float , x.size -1 , , self.weight, self.bias return y.to x.dtype norm = FP32LayerNorm 8 x = torch.randn 2, 8, dtype=torch.float16 print norm x .dtype Do not indiscriminately change modules in a pretrained model: a FP16 linear layer may reject an FP32 activation, and a later whole-model to call would convert the selected norm back to FP16. Extra casts can also cost time. You should use the model’s supported mixed-precision path and validate outputs on representative inputs. Measuring the Result Precision changes should be evaluated along three axes: - Memory: Parameter storage, peak allocated memory, and KV cache capacity. - Performance: Prefill latency, decode latency, and throughput at realistic batch sizes and sequence lengths. - Quality: Perplexity, task metrics, and generation comparisons on representative prompts. You can refer to the benchmarking rules and code examples from the previous post https://machinelearningmastery.com/measuring-performance-of-transformer-inference/ : warm up the model, synchronize asynchronous devices, separate prefill from decode, and report the hardware and software configuration. Compare output logits in addition to decoded text when debugging. Greedy outputs can remain identical even when many logits have changed, or diverge after one close token decision. A precision format reduces memory only in proportion to the tensors converted. If weights are half of total runtime memory because the KV cache is large, halving weight storage cannot halve total memory. Likewise, lower-precision weights help decode most when weight movement is the bottleneck. Large-batch prefill may already be compute-bound. Further Reading Below are some resources you may find useful: - PyTorch documentation. Floating point types https://pytorch.org/docs/stable/tensor attributes.html torch.dtype . This page lists the floating-point dtypes exposed by PyTorch. - PyTorch documentation. Mixed precision numerical accuracy https://docs.pytorch.org/docs/stable/notes/numerical accuracy.html reduced-precision-reduction-for-fp16-and-bf16-gemms . This explains reduced-precision accumulation and numerical behavior in matrix multiplication. - PyTorch blog. BF16 on Intel Xeon Processors https://pytorch.org/blog/empowering-pytorch-on-intel-xeon-scalable-processors-with-bfloat16/ . This documents CPU BF16 operations and autocast support. - Paulius Micikevicius et al. 2022 . FP8 Formats for Deep Learning https://arxiv.org/abs/2209.05433 . This paper describes the E4M3 and E5M2 FP8 formats and their use in deep learning. - Open Compute Project. OCP Microscaling Formats https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf . This specification describes block-scaled low-precision formats, including microscaling formats used with very small element types. Summary In this post, you learned how floating-point formats trade dynamic range, precision, memory, and hardware efficiency. FP16 and BF16 both use 16 bits but behave differently because their exponent and significand fields differ. You also learned that computation may use a different accumulation dtype from storage and that selected operations may need higher precision.