# INT8 vs FP8 Quantization: Why LLM Activations Have Outliers, and Why Scaling Granularity Matters > Source: > Published: 2026-09-21 20:19:10+00:00 *Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. [Star us](https://github.com/HexmosTech/LiveReview/) to help devs discover the project, give it a try, and share your feedback to help improve the product.* A 70B-parameter model in FP16 needs roughly 140 GB just to store its weights. Put the same weights in 8-bit and you get roughly 70 GB. That sounds like a solved problem. It isn't. The interesting part of LLM quantization is not reducing 16 bits to 8 bits. It is deciding **which 8-bit numbers are allowed to represent which parts of the model**. This is where things get strange. A single activation can be 50x or 100x larger than its neighbors. If you use one INT8 scale for the whole tensor, that one value can determine the scale for thousands of ordinary values. Then researchers discovered something even more useful: these outliers are often concentrated in particular feature dimensions. That observation led to a sequence of ideas involving Tim Dettmers, Song Han's group at MIT, and engineers from NVIDIA, Intel, Arm and others: **INT8 → find the outliers → isolate them or move them → choose better scaling → eventually use floating-point 8-bit formats.** The important lesson for developers is that **bit width, number representation, and scaling granularity are three separate decisions.** Let's build the intuition from the ground up. Suppose you have this activation vector: ``` [-0.8, 0.3, -0.2, 0.7, 0.1, 50.0] ``` You want to represent it with signed INT8. INT8 gives you 256 possible bit patterns, usually treated as approximately: ``` -127 ... 0 ... +127 ``` A simple symmetric quantizer chooses ``` scale = max(abs(x)) / 127 ``` Here: ``` scale = 50 / 127 ≈ 0.394 ``` Every value gets rounded onto a grid separated by about 0.394. So: ``` php 0.1 / 0.394 ≈ 0.25 -> 0 0.3 / 0.394 ≈ 0.76 -> 1 -0.2 / 0.394 ≈ -0.51 -> -1 0.7 / 0.394 ≈ 1.78 -> 2 ``` After dequantization: ``` php 0.1 -> 0.0 0.3 -> 0.394 0.2 -> 0.394 0.7 -> 0.788 ``` The small values have become rather crude. Now imagine that `50.0` was absent. The scale becomes: ``` 0.8 / 127 ≈ 0.0063 ``` Suddenly the quantization grid is about **62x finer**. That is the fundamental problem with absmax quantization: One extreme value can consume the dynamic range that the other 99.9% of the tensor wanted to use. This is why "INT8" by itself tells you surprisingly little. You also need to ask: **What is the scaling granularity?** The simplest scheme is **per-tensor scaling**. You look at the entire tensor and compute one scale: ``` s = max(abs(X)) / 127 ``` Then: ``` X_int8 = round(X / s) X_hat = X_int8 * s ``` The hardware story is attractive. One tensor. One scale. One INT8 GEMM. Very little metadata. But LLM activations have a peculiar distribution. In 2022, Tim Dettmers, Mike Lewis, Younes Belkada and Luke Zettlemoyer published **LLM.int8()** and investigated what was causing ordinary INT8 quantization to fail as Transformer models grew. Their measurements of OPT models found that large-magnitude activation features emerged systematically as models became larger. The particularly memorable result was at the 6.7B scale. They reported roughly 150,000 outlier values per sequence, but those outliers were concentrated into only about **six feature dimensions** across the Transformer. Those dimensions represented only around 0.1% of the feature values, yet removing them badly damaged model behavior. This is an important observation. The problem was not simply: ``` "There are a few large numbers." ``` It was closer to: ``` "There are a few special dimensions that repeatedly produce large numbers, and the model actually uses them." ``` That distinction changes the engineering solution. Suppose the activation matrix is: ``` X = token 1: [ 0.2 0.3 0.1 60.0 ] token 2: [ 0.1 0.2 0.3 55.0 ] token 3: [ 0.3 0.1 0.2 49.0 ] ``` Column 4 is clearly behaving differently. With per-tensor scaling, the `60.0` controls the scale for everything. With **per-channel scaling**, every column gets its own scale: ``` php channel 1 -> based on max(0.2, 0.1, 0.3) channel 2 -> based on max(0.3, 0.2, 0.1) channel 3 -> based on max(0.1, 0.3, 0.2) channel 4 -> based on max(60, 55, 49) ``` Now the first three channels can use a much finer INT8 grid. So why not simply quantize activations per channel? Because the matrix multiplication is: ``` Y = XW ``` and the activation channels are the **reduction dimension** of the matrix multiplication. If you scale every input channel independently, your computation becomes something like: ``` Y = (X * channel_scales) W ``` The scale is now entangled with every multiplication contributing to each output. The problem is therefore partly numerical and partly architectural: **the quantization scheme you would like is not necessarily the quantization scheme your fast GEMM kernel wants.** This is one of the recurring themes of quantization: The numerically nicest scheme is not necessarily the computationally cheapest scheme. The SmoothQuant paper from Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth and Song Han made this tension particularly clear. For their experiments, activation quantization with finer channel granularity could preserve accuracy, while conventional INT8 GEMM implementations favored coarser activation scaling. The trick was to move the problem somewhere else. This is one of the nicest pieces of algebra in practical LLM optimization. Suppose: ``` Y = XW ``` Pick a diagonal matrix of per-channel scales: ``` S = diag(s1, s2, ..., sn) Y = XW = (X S^-1)(S W) ``` Nothing has changed mathematically. You have simply moved a scale factor from one side of the matrix multiplication to the other. Define: ``` X' = X S^-1 W' = S W X'W' = XW ``` Now suppose one activation channel contains huge values. Choose `sj` to be large. The corresponding activation channel gets divided by `sj`: ``` php large activation -> smaller activation ``` while the corresponding weight channel gets multiplied by `sj`: ``` php weight -> somewhat larger weight ``` Why is that useful? Because weights tend to be considerably easier to quantize than activations. SmoothQuant exploits this asymmetry. Its smoothing factor can be expressed approximately as: ``` sj = max(|Xj|)^alpha / max(|Wj|)^(1-alpha) ``` where `alpha` controls how much quantization difficulty gets moved toward the weights. At: ``` alpha = 0 ``` you move essentially none of the activation difficulty. ``` alpha = 1 ``` you push the problem aggressively toward the weights. A common starting point is: ``` alpha = 0.5 ``` which roughly balances the ranges in each channel. Consider a toy channel: ``` max activation = 100 max weight = 0.01 ``` With `alpha = 0.5`: ``` s = sqrt(100 / 0.01) = sqrt(10000) = 100 ``` So the transformed ranges become approximately: ``` activation: 100 / 100 = 1 weight: 0.01 * 100 = 1 ``` You have turned: ``` activation range = 100 weight range = 0.01 ``` into: ``` activation range ≈ 1 weight range ≈ 1 ``` The matrix multiplication still computes the same function. The distribution has simply been rearranged into a form that is friendlier to quantization. This is why SmoothQuant is more interesting than "use smaller numbers." It is an **algebraic transformation that changes where quantization error is paid**. The original paper reported up to 1.56x speedup and 2x memory reduction for their evaluated models while maintaining close accuracy to higher precision baselines. Dettmers' approach was conceptually different. Instead of trying to eliminate the outliers, LLM.int8() observed: ``` php 99.9%+ of the values -> ordinary INT8 computation tiny set of important dimensions -> higher precision ``` So the matrix multiplication is decomposed. Conceptually: ``` Y = X_outlier W_outlier + X_regular W_regular ``` The outlier dimensions are computed in FP16, while the rest use INT8. This is a very practical compromise. Suppose the hidden dimension is 4096 and only a handful of feature dimensions are problematic. You do not need to make all 4096 dimensions expensive just because six of them are troublesome. This is analogous to designing a network where one pathological flow gets special handling instead of upgrading the entire network. And there is an operational advantage: **you preserve the bulk of the INT8 computation.** The LLM.int8() paper reported that more than 99.9% of values could still participate in 8-bit multiplication while the problematic dimensions were handled at higher precision. That work was also an important moment historically. Before it, "8-bit inference" often sounded like a relatively straightforward compression exercise. The experience with billion-parameter Transformers showed that scaling the model changed the statistical behavior of the activations. Quantization became a problem about **understanding the model's internal structure**, not merely reducing storage. Now we get to FP8. INT8 gives you a fixed-point-like grid after scaling. FP8 is fundamentally different. Instead of using all eight bits to represent an integer, you split them into: ``` sign + exponent + mantissa ``` The 2022 FP8 proposal from Paulius Micikevicius and collaborators defined two formats: ``` E4M3 1 sign bit + 4 exponent bits + 3 mantissa bits E5M2 1 sign bit + 5 exponent bits + 2 mantissa bits ``` The tradeoff is exactly what you would expect: ``` php more exponent bits -> more range more mantissa bits -> more precision ``` A rough mental model is: ``` INT8: values lie on an approximately uniform grid FP8: values are distributed approximately logarithmically across orders of magnitude ``` Imagine the numbers: ``` 0.125 0.25 0.5 1 2 4 8 16 ``` A floating-point representation naturally gives you useful coverage across such scales. An integer representation needs a scale to move the whole grid around. That makes FP8 much better suited to distributions with wide dynamic range. But there is a subtle point: **FP8 does not eliminate scaling.** A common FP8 computation still looks conceptually like: ``` x_fp8 = FP8(x / scale) x_hat = FP8_value * scale ``` The format gives you a wider range structure inside the 8 bits, but the scale still determines which region of that format your tensor occupies. And there are different FP8 formats for different numerical jobs. E4M3 provides more mantissa precision and less range. E5M2 sacrifices mantissa precision for more exponent range. That makes them useful for different parts of training. A common arrangement is E4M3 for forward-pass values and E5M2 for gradients. This was the broader significance of the 2022 FP8 work: INT8 and FP8 were no longer simply two ways to store "small numbers." They represented two different numerical philosophies: ``` INT8: "Give me a scale, then use a uniform integer grid." FP8: "Give me a scale, then let the exponent encode dynamic range." ``` That distinction matters enormously for LLM activations. The useful answer is: **look at the whole inference system, not just the datatype.** There are at least four variables: ``` 1. Datatype INT8 vs FP8 vs FP16/BF16 2. Scaling granularity per-tensor vs per-token vs per-channel vs block-wise 3. Quantization location weights, activations, KV cache, or some combination 4. Hardware/kernel support what your actual GPU or CPU can execute efficiently ``` This produces a very different engineering decision than: ``` "INT8 is smaller, therefore INT8 is better." ``` Consider a 70B model. Very roughly: ``` FP16 weights: 70B * 2 bytes ≈ 140 GB INT8 / FP8 weights: 70B * 1 byte ≈ 70 GB ``` You have saved approximately: ``` 70 GB ``` of parameter memory. That can determine whether a model fits on a given machine. But total inference memory is closer to: ``` weights + KV cache + temporary activations + workspace + runtime overhead ``` So 70 GB of weights does not mean a model is going to fit comfortably into a 70 GB device. There is also an economic dimension. Suppose your deployment needs enough GPU memory for: ``` 5 x 80 GB GPUs ``` at FP16, primarily because the weights are too large. If an 8-bit representation reduces the weight footprint enough to move the deployment to: ``` 3 x 80 GB GPUs ``` the economic effect can be larger than the numerical effect. You have potentially removed: ``` 2 GPUs ``` from every replica. At scale, that changes: ``` GPU rental rack capacity power network bandwidth failure surface deployment density ``` But this is where benchmarking matters. If your hardware has highly optimized FP8 Tensor Core kernels and your INT8 path requires awkward conversions, FP8 may win despite both using exactly one byte per stored value. On another machine, INT8 may have better kernel support. Even within one datatype, the difference between: ``` per-tensor per-token per-channel block-wise ``` can change both accuracy and runtime. A useful profiling equation is therefore: ``` effective cost ≈ memory traffic + compute time + scaling overhead + kernel inefficiency + synchronization overhead ``` The cheapest-looking representation on paper can lose once all five terms are included. The interesting story of LLM quantization is not: ``` php 16 bits -> 8 bits ``` It is: ``` How do we spend the limited precision budget? ``` The history makes this progression clear. Dettmers and colleagues encountered systematic activation outliers and separated a tiny set of problematic dimensions from the bulk of the computation. Xiao, Lin, Seznec, Wu, Demouth and Han showed that the algebra of the matrix multiplication could be exploited to **move quantization difficulty from activations into weights**. Micikevicius and collaborators then pushed the industry toward a floating-point 8-bit representation that gave hardware a better numerical tradeoff for deep learning. For developers, the mental model I find most useful is: ``` INT8 vs FP8 | +-- What numbers can the format represent? | +-- What scale maps my tensor into that range? | +-- How many elements share that scale? | +-- Where do the outliers live? | +-- Can the hardware execute that representation efficiently? ``` Once you think this way, "quantize the model to 8-bit" stops being a single operation. It becomes a small numerical systems-design problem. And that is probably the more useful way to approach the next generation of LLM inference. **When you deploy an LLM, which tradeoff would you optimize first: numerical accuracy, GPU memory, or raw tokens/second?** Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down. I'm building **LiveReview**, a blast-radius aware AI code review built for your business-critical systems. Instead of presenting every diff with equal emphasis, **LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.** Spend code review effort where business risk is highest — not spread evenly across every diff. ⭐ Star it on GitHub: LiveReview is an AI code reviewer that scores every hunk of a diff by **blast radius**: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff. *LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.* | The exact math, not a black box | Visualize blast radius at a glance | Every factor that feeds the score | |---|---|---| **Here's the goal:** **Click below to try LiveReview with your codebase:**