{"slug": "int8-vs-fp8-quantization-why-llm-activations-have-outliers-and-why-scaling", "title": "INT8 vs FP8 Quantization: Why LLM Activations Have Outliers, and Why Scaling Granularity Matters", "summary": "Shrijith Venkatramana, an engineer building the LiveReview AI code review tool, published a technical explainer on LLM quantization showing that activation outliers concentrated in a handful of feature dimensions can consume the dynamic range of an entire tensor under per-tensor INT8 absmax scaling. Citing LLM.int8() research by Tim Dettmers and colleagues, which found roughly 150,000 outlier values per sequence concentrated in about six feature dimensions at the 6.7B scale, he argues that bit width, number representation, and scaling granularity are three separate engineering decisions.", "body_md": "*Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. [Star us](https://github.com/HexmosTech/LiveReview/) to help devs discover the project, give it a try, and share your feedback to help improve the product.*\n\nA 70B-parameter model in FP16 needs roughly 140 GB just to store its weights.\n\nPut the same weights in 8-bit and you get roughly 70 GB.\n\nThat sounds like a solved problem.\n\nIt isn't.\n\nThe interesting part of LLM quantization is not reducing 16 bits to 8 bits. It is deciding **which 8-bit numbers are allowed to represent which parts of the model**.\n\nThis is where things get strange.\n\nA single activation can be 50x or 100x larger than its neighbors. If you use one INT8 scale for the whole tensor, that one value can determine the scale for thousands of ordinary values.\n\nThen researchers discovered something even more useful: these outliers are often concentrated in particular feature dimensions.\n\nThat observation led to a sequence of ideas involving Tim Dettmers, Song Han's group at MIT, and engineers from NVIDIA, Intel, Arm and others:\n\n**INT8 → find the outliers → isolate them or move them → choose better scaling → eventually use floating-point 8-bit formats.**\n\nThe important lesson for developers is that **bit width, number representation, and scaling granularity are three separate decisions.**\n\nLet's build the intuition from the ground up.\n\nSuppose you have this activation vector:\n\n```\n[-0.8, 0.3, -0.2, 0.7, 0.1, 50.0]\n```\n\nYou want to represent it with signed INT8.\n\nINT8 gives you 256 possible bit patterns, usually treated as approximately:\n\n```\n-127 ... 0 ... +127\n```\n\nA simple symmetric quantizer chooses\n\n```\nscale = max(abs(x)) / 127\n```\n\nHere:\n\n```\nscale = 50 / 127\n      ≈ 0.394\n```\n\nEvery value gets rounded onto a grid separated by about 0.394.\n\nSo:\n\n``` php\n 0.1 / 0.394 ≈ 0.25  -> 0\n 0.3 / 0.394 ≈ 0.76  -> 1\n-0.2 / 0.394 ≈ -0.51 -> -1\n 0.7 / 0.394 ≈ 1.78  -> 2\n```\n\nAfter dequantization:\n\n``` php\n0.1 -> 0.0\n0.3 -> 0.394\n0.2 -> 0.394\n0.7 -> 0.788\n```\n\nThe small values have become rather crude.\n\nNow imagine that `50.0` was absent.\n\nThe scale becomes:\n\n```\n0.8 / 127 ≈ 0.0063\n```\n\nSuddenly the quantization grid is about **62x finer**.\n\nThat is the fundamental problem with absmax quantization:\n\nOne extreme value can consume the dynamic range that the other 99.9% of the tensor wanted to use.\n\nThis is why \"INT8\" by itself tells you surprisingly little.\n\nYou also need to ask:\n\n**What is the scaling granularity?**\n\nThe simplest scheme is **per-tensor scaling**.\n\nYou look at the entire tensor and compute one scale:\n\n```\ns = max(abs(X)) / 127\n```\n\nThen:\n\n```\nX_int8 = round(X / s)\nX_hat  = X_int8 * s\n```\n\nThe hardware story is attractive.\n\nOne tensor.\n\nOne scale.\n\nOne INT8 GEMM.\n\nVery little metadata.\n\nBut LLM activations have a peculiar distribution.\n\nIn 2022, Tim Dettmers, Mike Lewis, Younes Belkada and Luke Zettlemoyer published **LLM.int8()** and investigated what was causing ordinary INT8 quantization to fail as Transformer models grew.\n\nTheir measurements of OPT models found that large-magnitude activation features emerged systematically as models became larger.\n\nThe particularly memorable result was at the 6.7B scale.\n\nThey reported roughly 150,000 outlier values per sequence, but those outliers were concentrated into only about **six feature dimensions** across the Transformer. Those dimensions represented only around 0.1% of the feature values, yet removing them badly damaged model behavior.\n\nThis is an important observation.\n\nThe problem was not simply:\n\n```\n\"There are a few large numbers.\"\n```\n\nIt was closer to:\n\n```\n\"There are a few special dimensions that repeatedly produce\nlarge numbers, and the model actually uses them.\"\n```\n\nThat distinction changes the engineering solution.\n\nSuppose the activation matrix is:\n\n```\nX =\n\ntoken 1: [ 0.2   0.3   0.1   60.0 ]\ntoken 2: [ 0.1   0.2   0.3   55.0 ]\ntoken 3: [ 0.3   0.1   0.2   49.0 ]\n```\n\nColumn 4 is clearly behaving differently.\n\nWith per-tensor scaling, the `60.0` controls the scale for everything.\n\nWith **per-channel scaling**, every column gets its own scale:\n\n``` php\nchannel 1 -> based on max(0.2, 0.1, 0.3)\nchannel 2 -> based on max(0.3, 0.2, 0.1)\nchannel 3 -> based on max(0.1, 0.3, 0.2)\nchannel 4 -> based on max(60, 55, 49)\n```\n\nNow the first three channels can use a much finer INT8 grid.\n\nSo why not simply quantize activations per channel?\n\nBecause the matrix multiplication is:\n\n```\nY = XW\n```\n\nand the activation channels are the **reduction dimension** of the matrix multiplication.\n\nIf you scale every input channel independently, your computation becomes something like:\n\n```\nY = (X * channel_scales) W\n```\n\nThe scale is now entangled with every multiplication contributing to each output.\n\nThe problem is therefore partly numerical and partly architectural:\n\n**the quantization scheme you would like is not necessarily the quantization scheme your fast GEMM kernel wants.**\n\nThis is one of the recurring themes of quantization:\n\nThe numerically nicest scheme is not necessarily the computationally cheapest scheme.\n\nThe SmoothQuant paper from Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth and Song Han made this tension particularly clear.\n\nFor their experiments, activation quantization with finer channel granularity could preserve accuracy, while conventional INT8 GEMM implementations favored coarser activation scaling.\n\nThe trick was to move the problem somewhere else.\n\nThis is one of the nicest pieces of algebra in practical LLM optimization.\n\nSuppose:\n\n```\nY = XW\n```\n\nPick a diagonal matrix of per-channel scales:\n\n```\nS = diag(s1, s2, ..., sn)\nY = XW\n  = (X S^-1)(S W)\n```\n\nNothing has changed mathematically.\n\nYou have simply moved a scale factor from one side of the matrix multiplication to the other.\n\nDefine:\n\n```\nX' = X S^-1\nW' = S W\nX'W' = XW\n```\n\nNow suppose one activation channel contains huge values.\n\nChoose `sj` to be large.\n\nThe corresponding activation channel gets divided by `sj`:\n\n``` php\nlarge activation -> smaller activation\n```\n\nwhile the corresponding weight channel gets multiplied by `sj`:\n\n``` php\nweight -> somewhat larger weight\n```\n\nWhy is that useful?\n\nBecause weights tend to be considerably easier to quantize than activations.\n\nSmoothQuant exploits this asymmetry.\n\nIts smoothing factor can be expressed approximately as:\n\n```\nsj = max(|Xj|)^alpha / max(|Wj|)^(1-alpha)\n```\n\nwhere `alpha` controls how much quantization difficulty gets moved toward the weights.\n\nAt:\n\n```\nalpha = 0\n```\n\nyou move essentially none of the activation difficulty.\n\n```\nalpha = 1\n```\n\nyou push the problem aggressively toward the weights.\n\nA common starting point is:\n\n```\nalpha = 0.5\n```\n\nwhich roughly balances the ranges in each channel.\n\nConsider a toy channel:\n\n```\nmax activation = 100\nmax weight     = 0.01\n```\n\nWith `alpha = 0.5`:\n\n```\ns = sqrt(100 / 0.01)\n  = sqrt(10000)\n  = 100\n```\n\nSo the transformed ranges become approximately:\n\n```\nactivation: 100 / 100 = 1\nweight:       0.01 * 100 = 1\n```\n\nYou have turned:\n\n```\nactivation range = 100\nweight range     = 0.01\n```\n\ninto:\n\n```\nactivation range ≈ 1\nweight range     ≈ 1\n```\n\nThe matrix multiplication still computes the same function.\n\nThe distribution has simply been rearranged into a form that is friendlier to quantization.\n\nThis is why SmoothQuant is more interesting than \"use smaller numbers.\"\n\nIt is an **algebraic transformation that changes where quantization error is paid**.\n\nThe original paper reported up to 1.56x speedup and 2x memory reduction for their evaluated models while maintaining close accuracy to higher precision baselines.\n\nDettmers' approach was conceptually different.\n\nInstead of trying to eliminate the outliers, LLM.int8() observed:\n\n``` php\n99.9%+ of the values -> ordinary INT8 computation\ntiny set of important dimensions -> higher precision\n```\n\nSo the matrix multiplication is decomposed.\n\nConceptually:\n\n```\nY = X_outlier W_outlier\n  + X_regular W_regular\n```\n\nThe outlier dimensions are computed in FP16, while the rest use INT8.\n\nThis is a very practical compromise.\n\nSuppose the hidden dimension is 4096 and only a handful of feature dimensions are problematic.\n\nYou do not need to make all 4096 dimensions expensive just because six of them are troublesome.\n\nThis is analogous to designing a network where one pathological flow gets special handling instead of upgrading the entire network.\n\nAnd there is an operational advantage:\n\n**you preserve the bulk of the INT8 computation.**\n\nThe LLM.int8() paper reported that more than 99.9% of values could still participate in 8-bit multiplication while the problematic dimensions were handled at higher precision.\n\nThat work was also an important moment historically.\n\nBefore it, \"8-bit inference\" often sounded like a relatively straightforward compression exercise.\n\nThe experience with billion-parameter Transformers showed that scaling the model changed the statistical behavior of the activations.\n\nQuantization became a problem about **understanding the model's internal structure**, not merely reducing storage.\n\nNow we get to FP8.\n\nINT8 gives you a fixed-point-like grid after scaling.\n\nFP8 is fundamentally different.\n\nInstead of using all eight bits to represent an integer, you split them into:\n\n```\nsign + exponent + mantissa\n```\n\nThe 2022 FP8 proposal from Paulius Micikevicius and collaborators defined two formats:\n\n```\nE4M3\n1 sign bit + 4 exponent bits + 3 mantissa bits\n\nE5M2\n1 sign bit + 5 exponent bits + 2 mantissa bits\n```\n\nThe tradeoff is exactly what you would expect:\n\n``` php\nmore exponent bits -> more range\nmore mantissa bits  -> more precision\n```\n\nA rough mental model is:\n\n```\nINT8:\nvalues lie on an approximately uniform grid\n\nFP8:\nvalues are distributed approximately logarithmically\nacross orders of magnitude\n```\n\nImagine the numbers:\n\n```\n0.125\n0.25\n0.5\n1\n2\n4\n8\n16\n```\n\nA floating-point representation naturally gives you useful coverage across such scales.\n\nAn integer representation needs a scale to move the whole grid around.\n\nThat makes FP8 much better suited to distributions with wide dynamic range.\n\nBut there is a subtle point:\n\n**FP8 does not eliminate scaling.**\n\nA common FP8 computation still looks conceptually like:\n\n```\nx_fp8 = FP8(x / scale)\nx_hat = FP8_value * scale\n```\n\nThe format gives you a wider range structure inside the 8 bits, but the scale still determines which region of that format your tensor occupies.\n\nAnd there are different FP8 formats for different numerical jobs.\n\nE4M3 provides more mantissa precision and less range.\n\nE5M2 sacrifices mantissa precision for more exponent range.\n\nThat makes them useful for different parts of training. A common arrangement is E4M3 for forward-pass values and E5M2 for gradients.\n\nThis was the broader significance of the 2022 FP8 work: INT8 and FP8 were no longer simply two ways to store \"small numbers.\"\n\nThey represented two different numerical philosophies:\n\n```\nINT8:\n\"Give me a scale, then use a uniform integer grid.\"\n\nFP8:\n\"Give me a scale, then let the exponent encode dynamic range.\"\n```\n\nThat distinction matters enormously for LLM activations.\n\nThe useful answer is: **look at the whole inference system, not just the datatype.**\n\nThere are at least four variables:\n\n```\n1. Datatype\n   INT8 vs FP8 vs FP16/BF16\n\n2. Scaling granularity\n   per-tensor vs per-token vs per-channel vs block-wise\n\n3. Quantization location\n   weights, activations, KV cache, or some combination\n\n4. Hardware/kernel support\n   what your actual GPU or CPU can execute efficiently\n```\n\nThis produces a very different engineering decision than:\n\n```\n\"INT8 is smaller, therefore INT8 is better.\"\n```\n\nConsider a 70B model.\n\nVery roughly:\n\n```\nFP16 weights:\n70B * 2 bytes\n≈ 140 GB\n\nINT8 / FP8 weights:\n70B * 1 byte\n≈ 70 GB\n```\n\nYou have saved approximately:\n\n```\n70 GB\n```\n\nof parameter memory.\n\nThat can determine whether a model fits on a given machine.\n\nBut total inference memory is closer to:\n\n```\nweights\n+ KV cache\n+ temporary activations\n+ workspace\n+ runtime overhead\n```\n\nSo 70 GB of weights does not mean a model is going to fit comfortably into a 70 GB device.\n\nThere is also an economic dimension.\n\nSuppose your deployment needs enough GPU memory for:\n\n```\n5 x 80 GB GPUs\n```\n\nat FP16, primarily because the weights are too large.\n\nIf an 8-bit representation reduces the weight footprint enough to move the deployment to:\n\n```\n3 x 80 GB GPUs\n```\n\nthe economic effect can be larger than the numerical effect.\n\nYou have potentially removed:\n\n```\n2 GPUs\n```\n\nfrom every replica.\n\nAt scale, that changes:\n\n```\nGPU rental\nrack capacity\npower\nnetwork bandwidth\nfailure surface\ndeployment density\n```\n\nBut this is where benchmarking matters.\n\nIf your hardware has highly optimized FP8 Tensor Core kernels and your INT8 path requires awkward conversions, FP8 may win despite both using exactly one byte per stored value.\n\nOn another machine, INT8 may have better kernel support.\n\nEven within one datatype, the difference between:\n\n```\nper-tensor\nper-token\nper-channel\nblock-wise\n```\n\ncan change both accuracy and runtime.\n\nA useful profiling equation is therefore:\n\n```\neffective cost\n≈ memory traffic\n+ compute time\n+ scaling overhead\n+ kernel inefficiency\n+ synchronization overhead\n```\n\nThe cheapest-looking representation on paper can lose once all five terms are included.\n\nThe interesting story of LLM quantization is not:\n\n``` php\n16 bits -> 8 bits\n```\n\nIt is:\n\n```\nHow do we spend the limited precision budget?\n```\n\nThe history makes this progression clear.\n\nDettmers and colleagues encountered systematic activation outliers and separated a tiny set of problematic dimensions from the bulk of the computation.\n\nXiao, Lin, Seznec, Wu, Demouth and Han showed that the algebra of the matrix multiplication could be exploited to **move quantization difficulty from activations into weights**.\n\nMicikevicius and collaborators then pushed the industry toward a floating-point 8-bit representation that gave hardware a better numerical tradeoff for deep learning.\n\nFor developers, the mental model I find most useful is:\n\n```\nINT8 vs FP8\n        |\n        +-- What numbers can the format represent?\n        |\n        +-- What scale maps my tensor into that range?\n        |\n        +-- How many elements share that scale?\n        |\n        +-- Where do the outliers live?\n        |\n        +-- Can the hardware execute that representation efficiently?\n```\n\nOnce you think this way, \"quantize the model to 8-bit\" stops being a single operation.\n\nIt becomes a small numerical systems-design problem.\n\nAnd that is probably the more useful way to approach the next generation of LLM inference.\n\n**When you deploy an LLM, which tradeoff would you optimize first: numerical accuracy, GPU memory, or raw tokens/second?**\n\nYour team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.\n\nI'm building **LiveReview**, a blast-radius aware AI code review built for your business-critical systems.\n\nInstead of presenting every diff with equal emphasis, **LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.**\n\nSpend code review effort where business risk is highest — not spread evenly across every diff.\n\n⭐ Star it on GitHub: \n\nLiveReview is an AI code reviewer that scores every hunk of a diff by **blast radius**: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.\n\n*LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.*\n\n| The exact math, not a black box | Visualize blast radius at a glance | Every factor that feeds the score | \n|---|---|---|\n\n**Here's the goal:**\n\n**Click below to try LiveReview with your codebase:**", "url": "https://wpnews.pro/news/int8-vs-fp8-quantization-why-llm-activations-have-outliers-and-why-scaling", "canonical_source": "https://dev.to/shrsv/int8-vs-fp8-quantization-why-llm-activations-have-outliers-and-why-scaling-granularity-matters-2p3j", "published_at": "2026-09-21 20:19:10+00:00", "updated_at": "2026-09-21 20:54:38.715007+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research", "ai-infrastructure", "developer-tools"], "entities": ["Shrijith Venkatramana", "LiveReview", "HexmosTech", "Tim Dettmers", "LLM.int8()", "MIT", "NVIDIA", "Intel"], "alternates": {"html": "https://wpnews.pro/news/int8-vs-fp8-quantization-why-llm-activations-have-outliers-and-why-scaling", "markdown": "https://wpnews.pro/news/int8-vs-fp8-quantization-why-llm-activations-have-outliers-and-why-scaling.md", "text": "https://wpnews.pro/news/int8-vs-fp8-quantization-why-llm-activations-have-outliers-and-why-scaling.txt", "jsonld": "https://wpnews.pro/news/int8-vs-fp8-quantization-why-llm-activations-have-outliers-and-why-scaling.jsonld"}}