{"slug": "the-softmax-wall-and-the-2-4-sparse-attention-trap", "title": "The Softmax Wall and the 2:4 Sparse Attention Trap", "summary": "Zephyr's ZYFA release bypasses hardware exponential instructions with a bit-pattern trick called \"Lean P\" that computes exponentials in e4m3 using one floating-point multiply-add and one integer conversion, cutting FP8 softmax from 896 to 384 vector cycles and lifting steady-state matrix utilization on AMD's Instinct MI355X from 60% to 86%, according to the article. The analysis argues chipmakers have inverted the attention roofline by scaling matrix throughput per generation while leaving special function units and shared-memory bandwidth behind: on MI355X the two attention GEMMs for a tile take 1,024 cycles while the FP8 softmax takes 896 vector cycles, leaving the matrix pipe idle 31.7% of the time. On NVIDIA hardware, the MUFU.EX2 exponential unit still takes 1,024 cycles, forcing FlashAttention-4 to route 10%–25% of its exponentials onto FP32 FMA pipelines via degree-3 Sollya polynomial emulation, and Vera Rubin's second SFU doubling applies only to FP16 exponentials at 64 ops/clk/SM versus 32 for FP32.", "body_md": "## TL;DR[#](#tldr)\n\nFor the past decade, optimizing Transformer attention was treated as a synonym for accelerating matrix multiplication. Every hardware generation followed the same playbook: drop the numerical precision of the Tensor Cores or Matrix Cores—from FP32 to FP16/BF16, then to FP8, and now to 4-bit MXFP4 and NVFP4—to double the matrix FLOPs per clock.\n\nThat playbook just hit a physical wall of its own making. **By scaling matrix throughput  to  per generation while leaving Special Function Units (SFUs), vector ALUs, and shared-memory bandwidth behind, chipmakers inverted the attention roofline.**\n\nIn FP8 and FP4, attention is no longer bottlenecked by matrix multiplication. It is bottlenecked by the **transcendental and vector math of softmax** and the **on-chip memory traffic** required to shuttle intermediate scores between execution domains:\n\n- **On AMD’s Instinct MI355X (`gfx950`, CDNA 4) and Instinct MI455X (` gfx1250`, CDNA 5):** On MI355X, dropping from BF16 to FP8 doubles matrix throughput, so the two attention GEMMs ( and ) for a  tile take just**1,024 cycles** . The exact FP8 softmax between them, however, runs on the vector ALU at unchanged speed and takes**896 vector cycles** —leaving AMD’s default AITER kernel with its matrix pipe**idle 31.7% of the time** . To break that bottleneck in**ZYFA** (released this week), Zyphra had to bypass the hardware exponential instruction altogether with**“Lean P”** —an IEEE/FP8 bit-pattern trick that computes  in`e4m3` using**one floating-point multiply-add and one integer conversion** , cutting softmax from**896 to 384 cycles** and lifting steady-state matrix utilization from**60% to 86%** . Even then, preserving dynamic range in FP8 probabilities flips more operand bits in the multiplier trees, triggering a switching-power clock drop from**2,108 MHz to 1,628 MHz** on real Mistral-7B activations. In**CDNA 5 (`gfx1250`, MI455X)** , AMD overhauled the compute core—switching to Wave32`WMMA` , expanding addressable VGPRs  to 1,024 per wave, adding`VOPD3` dual-issue and hardware block-scaled conversions (`V_CVT_SCALEF32` ), and**doubling transcendental throughput ( vs. MI355X)** . Yet because CDNA 5 simultaneously scales OCP MXFP8 and MXFP4 matrix throughput by **over MI355X** (reaching**20.13 PFLOPS** in FP8/MXFP8 and**40.26 PFLOPS** in MXFP4 against**5.03 PFLOPS** in BF16), low-bit matrix engines on MI455X outpace the doubled transcendental units by an even wider margin.\n- **On NVIDIA’s Blackwell (`sm_100`), Blackwell Ultra (` sm_103`), and Vera Rubin (` sm_107`):** From Hopper H100 to Blackwell B200, BF16 Tensor Core throughput per SM doubled (), while the multifunction exponential unit (`MUFU.EX2` ) stayed frozen at`MUFU.EX2` takes**1,024 cycles** —already tying BF16 matmuls (), taking **longer** than FP8 matmuls (), and taking **longer** than NVFP4 matmuls ().**FlashAttention-4** had to route**10%–25%** of its exponentials onto general-purpose FP32 FMA pipelines via degree-3 Sollya polynomial emulation and skip ~90% of online softmax rescaling steps. NVIDIA responded by doubling SFU throughput in**Blackwell Ultra (B300)** to .**Vera Rubin (`sm_107`) did not double it again for FP32.** NVIDIA’s own Rubin spec table lists`EX2` at**32 ops/clk/SM for FP32 and 64 for FP16** , so the second doubling only exists for half-precision exponentials. Rubin meanwhile doubled per-SM FP8 Tensor Core throughput. With an FP32 exponential, a Rubin FP8 tile is **SFU-bound** and an NVFP4 tile**Rubin CPX** context GPU with “3× exponent operations,” was pulled from the roadmap at GTC 2026.\n\nAgainst this inverted roofline, a second hardware pitch has resurfaced: **accelerating dynamic attention with fine-grained 2:4 Sparse Tensor Cores**—most prominently in NVIDIA’s **Vera Rubin (`sm_107`)**, which adds the PTX 9.4 instruction **` tcgen05.ld.red.spcompress`** to prune 2 of every 4 attention logits while loading from Tensor Memory (TMEM) so that  can run on `tcgen05.mma.sp`, alongside 4:2 structured-sparse matrix instructions (` V_SWMMAC`) in AMD’s **CDNA 5 (` gfx1250`)**. The headline numbers are still sold with sparsity. Rubin’s 50 PF NVFP4 “inference” figure is Transformer Engine compute with a new, undisclosed “adaptive compression” scheme that SemiAnalysis says is neither 2:4 nor Blackwell’s pairwise 4:8; the dense training figure is 35 PF. AMD’s MI455X datasheet lists 2:4 peaks only for FP16, BF16 and INT8, at 10.07 PF, which is half of the same chip’s 20.13 PF of dense FP8.\n\n**Applying 2:4 (or FP4 pairwise 4:8) hardware sparse matrix units to attention is an architectural trap**, for three reasons.\n\n1. **It accelerates the shrinking part of the loop while dumping repack and online-pruning work onto the bottleneck.** On Blackwell and Rubin’s`tcgen05` architecture, Tensor Cores write  into**TMEM** and require operand A () for  to come from**TMEM or Shared Memory (SMEM)** —never from thread registers. Yet Rubin’s`tcgen05.ld.red.spcompress` writes its compressed FP32 values (`cdata` ) and 2-bit metadata (`mdata` ) into**general-purpose registers** . Software must still execute sparse softmax in registers, downcast dtypes, repack  into the MMA A-layout, repack and swizzle the 2-bit metadata, store both back to SMEM/TMEM (`tcgen05.st` ), and wait on a completion fence before`tcgen05.mma.sp` can issue. (AMD’s CDNA 5 avoids that TMEM round-trip because`V_SWMMAC` keeps operands and indices in VGPRs, but CDNA 5 lacks a hardware`spcompress` instruction, leaves unscaled FP8 at the same  tile depth as dense`V_WMMA` , and disables structured sparsity entirely on block-scaled`V_WMMA_SCALE*` instructions.) Across Ada, Hopper, and Blackwell, online pruning and compression of  consumes**70% to 87% of total runtime** , turning a  standalone FP16 sparse-GEMM speedup into a net **slowdown** in FP16 (and trailing dense NVFP4 at2. **At equal bits and equal Tensor Core tiers, dropping one dense precision step crushes 2:4 on both speed and accuracy.** Even before counting metadata or 2:4 structural constraints, analytical distribution models and convex SDP lower bounds show that uniform quantization dominates unstructured pruning at every practical bit-width above 3 bits. Once 2:4’s 2-bit index per kept value ( per original element) is counted, 2:4 on -bit values costs the exact same memory as dense -bit quantization (). Quantization error energy shrinks  with every added bit, whereas 2:4 deletes a fixed fraction of tensor energy ( on Gaussian data) that never shrinks with bit-width—and Hadamard rotations () make 2:4 twice as bad by spreading energy evenly across each 4-tuple. Across 221,408 measured configurations, dense FP8 and dense NVFP4 beat 2:4 at the  and  compute tiers on every operand.\n3. **Perplexity hides severe degradation in fact recall and multi-step reasoning, single-sided pruning caps speedup at , and Blackwell/Rubin’s pairwise 4:8 FP4 constraint collapses model quality.** Evaluating pruned models by WikiText-2 perplexity is a mirage: parametric fact recall breaks at just**20%–30% sparsity** , pruning small weights monotonically and irreversibly impairs harder tasks by destroying the pre-trained loss basin, and on Qwen3-1.7B, 2:4 sparsity collapses reasoning accuracy from**54.0% to 15.3%** . Trying to rescue 2:4 accuracy by relaxing to 6:8 sparsity (*SlideSparse* ) forces a  activation duplication tax that caps theoretical speedup at . Inside attention, pruning  alone leaves  dense ( Amdahl ceiling), pruning  or  wrecks early layers and long-context retrieval, and at the NVFP4 floor hardware forces**pairwise 4:8 sparsity** —blowing up one-shot Llama-3.1-8B perplexity from  to\n\nYou can see the verdict in the production serving codebases. In 2026, **vLLM** and **LLM Compressor** deleted their 2:4 sparsity kernels for lack of adoption. Where fine-grained sparsity genuinely pays off—such as  **unstructured activation sparsity** in ReLU-gated FFNs via Sakana AI and NVIDIA’s **Tile-wise ELLPACK (`TwELL`)** kernels on CUDA Cores, or static weights combined with low-rank adapters below the 4-bit floor—it works by skipping  of column/row fetches or operating offline on static weights, neither of which applies to 2:4 attention. In attention, **vLLM v0.33.0**, **SGLang v0.5.23**, and **FlashInfer v0.7.3**—across NVIDIA Hopper/Blackwell/Rubin, AMD MI355X/MI455X, and Google TPUv7 SparseCores—converged on **coarse token-level sparse attention** (**DeepSeek Sparse Attention / DSA**, **QSA**, **MSA**): a lightweight `ReLU` indexer selects the top  tokens out of , skipping **99.8% of the  matmuls, 99.8% of the softmax exponentials, and 99.8% of the KV HBM reads** before the main attention loop even starts, and then executes the surviving tokens as **dense tiles** at peak low-precision Tensor Core speed.\n\n## 1. Why did FP8 and FP4 make the softmax the bottleneck?[#](#1-why-did-fp8-and-fp4-make-the-softmax-the-bottleneck)\n\nTo see why softmax suddenly became the governor of long-context prefill and training, start from the two-stage arithmetic structure of self-attention. Given query, key, and value matrices for an attention head of dimension , tiled into blocks of queries by keys, each iteration of the inner FlashAttention loop executes two matrix multiply-accumulates (MMAs) separated by a row-wise online softmax:\n\nCount the operations required for a single tile with head dimension :\n\n1. **Matrix multiply-accumulate work ( and ):** Each of the two GEMMs performs  multiply-adds, for a total of\n runs on dedicated matrix engines (NVIDIA Tensor Cores, AMD Matrix Cores) whose per-cycle throughput**doubles (or quadruples) every time you halve the operand bit-width** —from 16-bit (`BF16` /`FP16` ) to 8-bit (`FP8 E4M3` /`MXFP8` ) to 4-bit (`NVFP4` /`MXFP4` ).\n2. **Non-matmul softmax and rescaling work ( and  correction):** Between the two GEMMs, every one of the  score elements in  must undergo a row-max reduction, a scaled subtraction, a base-2 transcendental exponentiation\na row-sum reduction , a down-conversion to the MMA input format (`BF16` ,`FP8` , or`FP4` ), and—whenever a block’s row maximum increases—a vector rescaling of the  running output accumulator .\n\nThe two terms scale differently:\n\n- **The ratio of matrix FLOPs to transcendental exponentials is fixed solely by the head dimension :** For the standard head dimension  used across Llama, Qwen, DeepSeek, Kimi, and GLM,**every 1 exponential evaluation is paired with exactly  matrix FLOPs** (and for , as in Meta’s 816B Muse Spark Open, only**256 matrix FLOPs per exponential** ).\n- **Lowering GEMM precision to FP8 or FP4 shrinks the cycle time of those 512 matrix FLOPs by  to , but does not reduce  by a single operation.** Because attention logits  must be accumulated and exponentiated in FP32 to avoid catastrophic overflow or underflow before normalization, the softmax stage still has to evaluate  exponentials in FP32 on Special Function Units (NVIDIA’s`MUFU.EX2` ) or vector ALUs (AMD’s VALU`v_exp_f32` ), and still has to move , , and  across on-chip memories (TMEM, SMEM/LDS, and the register file).\n\nFor an attention kernel to remain compute-bound on its matrix engines without stalling on softmax, the hardware’s per-SM/per-WGP matrix throughput and exponential throughput must satisfy:\n\nAt , **the moment a chip’s matrix throughput exceeds  its exponential throughput (), the exponential unit takes more clock cycles per tile than both matrix multiplications combined.** At , that crossover happens at .\n\n### 1.1 How did the cycle budget change from Hopper (`sm_90`) to Blackwell (` sm_100`)?[#](#11-how-did-the-cycle-budget-change-from-hopper-sm_90-to-blackwell-sm_100)\n\nNow plug the actual per-SM hardware throughputs of NVIDIA’s Hopper H100 (`sm_90`) and Blackwell B200 / GB200 (` sm_100`) into that inequality, following the roofline analysis in **FlashAttention-4**:\n\n1. \n**On Hopper H100 (`sm_90`):**\n  - **BF16 Tensor Cores:** ( dense across 132 SMs).\n  - **FP8 Tensor Cores:** ( dense).\n  - **Exponential unit (`MUFU.EX2`):** .\n  - **Shared Memory (SMEM) read bandwidth:** .\n  - Check the ratio at : in BF16, . For a  tile, the two BF16 MMAs take , while the  exponentials take . Because matrix math took  longer than `MUFU.EX2` , FlashAttention-3 could hide the entire BF16 softmax behind the Tensor Cores using a two-warpgroup ping-pong schedule. In**FP8** on Hopper, however, :  dropped to , already tying .\n2. \n**On Blackwell B200 / GB200 (`sm_100`):** As zartbot’s November 2025 microarchitecture analysis and the FlashAttention-4 roofline model both show, NVIDIA aggressively expanded the 5th-generation Tensor Core (`tcgen05.mma` ) and added 256 KB of Tensor Memory (TMEM) plus DSMEM per SM—consuming so much die area that the SM count per die was trimmed to 80 (148 enabled across two dies on B200, 160 on GB200)**while the SFU (`MUFU.EX2`) and SMEM bandwidth were left unscaled** :\n  - **BF16 Tensor Cores:** Doubled to  - **FP8 Tensor Cores:** Doubled to  - **NVFP4 Tensor Cores:** Quadrupled to  - **Exponential unit (`MUFU.EX2`):** Still  - **Shared Memory (SMEM) read bandwidth:** Still\n\nCompute the exact cycle budget per SM on B200 (`sm_100`) for a  forward tile ():\n\n- **MMA compute cycles:**\n- **SMEM read cycles (BF16 operands):** is a shared-shared (`SS` ) MMA reading  of  and  of  from SMEM;  is a tensor-shared (`TS` ) MMA reading  directly from TMEM and  of  from SMEM. At 2 bytes/element and :\n- **Exponential unit (`MUFU.EX2`) cycles:**\n\nAnd in the **backward pass** on B200 (which chains 5 MMAs: , , , , and ), 5 BF16 MMAs take , while 1-CTA SMEM traffic (reading 8 BF16 MMA operands from SMEM = , writing BF16  to SMEM = , and writing + TMA-reading FP32  = ) takes **—exceeding Tensor Core compute time by** **30%** (and requiring 2-CTA cooperative MMA with  of DSMEM  exchange to bring total SMEM traffic down to **).**\n\nSo on the B200 forward pass:\n\n- Even in **BF16** , : the SFU () is already tied 1:1 with the Tensor Cores (), before counting a single cycle of row-max reduction, row-sum reduction, scale subtraction, or TMEM load/store.\n- In **FP8** , : the SFU () takes **longer** than both Tensor Core GEMMs combined ().\n- In **NVFP4** , : the SFU () takes **longer** than both Tensor Core GEMMs combined ().\n\n### 1.2 Did B300 (`sm_103`) and Rubin (` sm_107`) fix the exponential unit?[#](#12-did-b300-sm_103-and-rubin-sm_107-fix-the-exponential-unit)\n\nAt this point, a hardware defender will raise the natural objection: *“NVIDIA recognized the B200 SFU mistake. Blackwell Ultra (B300 / GB300, `sm_103`) doubled SFU throughput from 16 to 32 ops/clk/SM, and Vera Rubin (`sm_107`) doubles it again.”*\n\nThe first half is true. The second half is true only for half precision. Check both against NVIDIA’s own specifications and benchmarks.\n\nIn **Blackwell Ultra (B300 / GB300, `sm_103`)**, NVIDIA doubled `MUFU.EX2` throughput from **16 to 32 ops/clk/SM**. In NVIDIA’s `mufu_ex2_bench` microbenchmark, isolating raw `MUFU.EX2` loops across the full GPU confirms the  jump:\n\n- **GB200 (`sm_100`):** (` BF16x2` , ),  (`BF16` ), and  (`FP32` ).\n- **GB300 (`sm_103`):** (` BF16x2` , ),  (`BF16` ), and  (`FP32` ).\n\nOn DeepSeek-V3 forward propagation (FPROP) in FP8, that single change—doubling `MUFU.EX2` without changing FP8 Tensor Core FLOPs—boosted attention FPROP throughput by **~35%**. That 35% jump is the clearest possible proof of how badly B200 was SFU-starved in FP8.\n\n**What 32 `MUFU.EX2` ops/clk/SM on B300 buys against the precision ladder:**\n\n- For a  tile on B300 (`sm_103` ), .\n- In **BF16** (), B300 merely restores the 2:1 MMA-to-SFU cushion that Hopper H100 already had in 2022.\n- In **FP8** (),  is a**zero-margin 512-vs.-512 tie** : the SFU must stay 100% utilized on every single clock cycle just to keep the FP8 Tensor Cores from stalling, leaving zero slack for row-max, row-sum, or`tcgen05.ld` /`tcgen05.st` latency.\n- In **NVFP4** —where B300`sm_103` also widened dense Tensor Core reduction depth from  to , pushing dense NVFP4 from  to**NVFP4 Tensor Cores still outrun the doubled SFU by  to .**\n\nNow **Vera Rubin (`sm_107`)**. Start with what Rubin did to the Tensor Cores:\n\n- By moving PCIe, NVLink 6, and NVLink-C2C off the compute dies onto separate I/O dies connected via NV-HBI, Rubin expands the SM count by  (from 160 SMs on B300 to **224 SMs** on Rubin, with 896 Tensor Cores) and raises clock frequency by .\n- **Rubin doubles the per-cycle -dimension throughput of the Tensor Core for FP8** (and expands NVFP4 from  to ). Across the GPU, dense BF16 rises from  (B300) to **per Tensor Core per clock** ; NVIDIA’s rack page gives 1.26 EF FP8 training for 72 GPUs), and NVFP4 reaches **dense training /  inference** .\n\nNVIDIA’s own Rubin compute table lists “Softmax acceleration (SFU EX2 Ops/Clk/SM for FP32 | FP16)” as **16** for Blackwell and **32 | 64** for Rubin. B300 already runs FP32 `MUFU.EX2` at 32 per clock per SM (the GB300 `mufu_ex2_bench` numbers above). So Rubin’s FP32 exponential rate per SM is **the same as B300’s**. The new doubling is the FP16 `EX2` path.\n\nDo the per-SM per-clock arithmetic for a tile on Rubin. Per-SM FP8 throughput doubles to , so . NVFP4 reaches , so . Against that:\n\n- **With an FP32 exponential** (32 per clock): . FP8 is now **SFU-bound** and NVFP4- **With an FP16 exponential** (64 per clock): . FP8 returns to a zero-slack  tie and NVFP4 stays\n\nSo the second doubling helps only kernels that move the exponential itself to half precision. That is a numerics decision, not a free speedup. The scaled logit and its exponential then carry FP16’s 10-bit mantissa and narrower range instead of FP32’s, before is rounded again to BF16, FP8 or FP4 for the MMA. It is the same kind of trade ZYFA’s Lean P makes in software on AMD (§2.2): precision that the destination format will throw away anyway is spent on throughput instead. On Rubin, NVIDIA built that trade into the SFU and left the FP32 path where B300 had it.\n\n#### 1.2.1 What about the vector units, Rubin CPX, and Rubin Ultra?\n\n**The software escape hatch is narrowing too.** FA4’s way around the SFU (§2.1) moves 10%–25% of the exponentials onto FP32 FMA pipes. NVIDIA’s same Rubin blog publishes the FP32 vector rates: **67 TFLOPS (Hopper), 80 (Blackwell), 130 (Rubin)**. Against dense FP8 matrix throughput of roughly 2 PF (H100), 4.5 PF (B200) and 17.5 PF (Rubin), the ratio of FP8 matrix FLOPs to FP32 vector FLOPs goes from about **30:1 to 56:1 to 135:1**. From Blackwell to Rubin the FMA pipes grew  while FP8 matrix grew about . The idle FMA capacity that FA4 borrows shrinks per matrix FLOP each generation, so polynomial emulation buys less on Rubin than it did on B200.\n\n**Rubin CPX was NVIDIA’s dedicated answer, and it was pulled.** In September 2025 NVIDIA announced Rubin CPX, a monolithic GDDR7 GPU for the context (prefill) phase with up to 30 PF of NVFP4 and “3x faster attention capabilities compared with NVIDIA GB300 NVL72 systems.” Third-party notes from the launch describe the mechanism as “3x exponent operations” in “attention acceleration cores.” That is a chip built around the softmax wall, for exactly the long-context prefill this article is about. It never shipped. CPX was absent from the six-chip Rubin launch at CES 2026 and from the GTC 2026 roadmap. Ian Buck told reporters “we’ve pulled CPX. It’s still a good idea,” and NVIDIA said prefill work goes back to the Rubin GPU, with Groq 3 LPX racks taking the decode-specialist slot. A revived CPX with 168 GB of HBM4 was reported in August 2026 by one analyst (Ming-Chi Kuo) and has not been confirmed. As shipped, the Rubin generation’s context processing runs on a GPU whose FP32 `EX2` rate per SM equals B300’s.\n\n**Rubin Ultra changes package size, not the per-SM ratio, as far as anything is published.** NVIDIA’s only official Rubin Ultra spec is from March 2025: four reticle-sized dies, 100 PF FP4 and 1 TB of HBM4E per package, for 2H 2027 in Kyber racks. In June 2026 SemiAnalysis reported the quad-die design was cut to two dies; NVIDIA has not confirmed that or published a replacement. Either way, NVIDIA has published no SM-level numbers for Rubin Ultra: no SFU rate, no per-SM Tensor Core rate. Adding dies multiplies matrix units and SFUs together. The tile ledger above only moves if the SM itself changes, so there is nothing yet to suggest Rubin Ultra fixes it.\n\n**A note on names**, since sources mix them:\n\n- **Rubin** is the shipping GPU (two dies, 224 SMs, 288 GB HBM4). SemiAnalysis calls it R200; “R100” appears in some early coverage but is not an NVIDIA product name.\n- **VR200 NVL144** was the GTC 2025 name for the Rubin rack, counting dies. NVIDIA went back to counting packages, so the same rack ships as**Vera Rubin NVL72** .\n- **Rubin Ultra** (R300 in some trackers) pairs with**Kyber** . “NVL576” meant one Kyber rack counted by dies in 2025; in the GTC 2026 roadmap, Kyber NVL144 is one rack and Vera Rubin Ultra NVL576 is eight racks.\n- **Feynman** (2028) has no published performance numbers.\n\n### 1.3 Does AMD hit the same wall on CDNA 4 (`gfx950`) and CDNA 5 (` gfx1250`)?[#](#13-does-amd-hit-the-same-wall-on-cdna-4-gfx950-and-cdna-5-gfx1250)\n\nThis asymmetric scaling wall is not an artifact of NVIDIA’s SM layout. It hits AMD’s Instinct line on both **CDNA 4 (`gfx950`, MI355X)** and the newly disclosed **CDNA 5 (` gfx1250`, MI455X / MI450)**—first through CDNA 4’s static wave-scheduling constraints, and then through CDNA 5’s  microscaling matrix jump.\n\n#### 1.3.1 Why does AITER leave the MI355X (`gfx950`) matrix pipe idle 31.7% of the time?\n\nOn MI355X (`gfx950`), each Compute Unit (CU) contains four Wave64 SIMD/Matrix Core quadrants. Moving from BF16 to FP8 `e4m3` doubles the throughput of the Matrix Fused Multiply-Add (`MFMA`) units. However, the softmax between  and  executes on the Vector ALU (`VALU`), whose throughput does not change when the GEMMs switch from BF16 to FP8.\n\nIn Zyphra’s **ZYFA** technical report (released this week for MI355X on ROCm 7.2.3), the authors traced the exact cycle breakdown for a single SIMD processing a tile of  **queries against  keys** at head dimension :\n\n- **FP8 Matrix Core work ( + ):**- **Exact FP8 Softmax work on the Vector ALU (VALU):**\n\nThat is  **of softmax against**  **of FP8 GEMM ( of the entire matrix budget).**\n\nOn paper, a naive roofline model would say: , so if you overlap VALU softmax for tile with MFMA matrix multiplication for tile , the 896 vector cycles should hide completely behind the 1,024 matrix cycles.\n\n**Why doesn’t that happen in silicon on MI355X?**\nBecause unlike NVIDIA’s `tcgen05` (which runs asynchronously in a separate TMEM domain and signals `mbarrier` completion in hardware), **MI355X does not track matrix instruction completion in hardware.** On CDNA 4, both MFMA instructions and VALU instructions are issued by the same wave schedulers and share the vector register file (split into 256 architectural VGPRs and 256 accumulation VGPRs), with no hardware scoreboard for asynchronous MFMA completion. Consequently, overlapping the 896 cycles of VALU softmax with the 1,024 cycles of MFMA requires **static instruction-order interleaving** across waves sharing a SIMD.\n\nBecause has a hard data dependency on the completion of the MFMA, and the MFMA has a hard data dependency on the completion of the softmax VALU instructions (plus cross-lane row-max and row-sum reductions and VGPR-to-LDS staging), of slack is far too small to absorb pipeline bubbles:\n\n- In **AITER** (AMD’s default production kernel library), Zyphra’s thread traces show that**the MI355X matrix pipe sits completely idle for 31.7% of cycles** , capping 32K-token FP8 attention at  (**39% of the MI355X FP8 datasheet peak** ) on  and just  on DeepSeek’s MLA shape ().\n- Even when Zyphra implemented a hand-scheduled **two-wave ping-pong schedule** (where two waves share each SIMD, one issuing MFMA instructions while the other computes softmax probabilities on the VALU, and then swapping), running**exact FP8 softmax (896 cycles)** only kept the matrix pipe busy for**60% of steady-state cycles** .\n\n#### 1.3.2 What changes on CDNA 5 (`gfx1250`, Instinct MI455X)?\n\nHow does AMD’s next-generation **CDNA 5 architecture** on the **Instinct MI455X / MI450 (`gfx1250`)**, described in AMD’s whitepaper and the preliminary ISA guide published in July 2026, change this equation?\n\nIn CDNA 5, AMD made the most sweeping microarchitectural shift since the original GCN/CDNA split, converging several RDNA 4 (`gfx12`) execution concepts with datacenter matrix and memory engines across 8 TSMC N2 Accelerator Complex Dies (XCDs):\n\n1. **From Wave64 CUs and `MFMA` to Wave32 WGPs and `WMMA`:**  - The 32 Compute Units per XCD of MI355X are replaced by **32 Work-Group Processors (WGPs)** per XCD (2 Shader Engines  16 WGPs per XCD =**256 active WGPs** chip-wide, running at up to**2,400 MHz** ).\n  - Each WGP (“Double Compute Unit”) contains **four 32-lane SIMD32 units** and four scalar ALUs, and**operates exclusively in Wave32 mode** (dropping Wave64 entirely), supporting up to**64 resident waves per WGP** (16 per SIMD32) and up to 32 workgroups per WGP.\n  - CDNA’s old Wave64 `MFMA` instructions are replaced by Wave32**Wave Matrix Multiply-Accumulate (`WMMA`)** instructions (` V_WMMA_*` and block-scaled`V_WMMA_SCALE*` ), operating on  output tiles (plus  for FP4).\n2. The 32 Compute Units per XCD of MI355X are replaced by \n3. **Transcendental and Vector Upgrades ( Throughput + Native BF16 & `TANH`):**  - Just as NVIDIA doubled `MUFU.EX2` in B300, AMD’s CDNA 5 whitepaper explicitly states that**existing transcendental operations have twice () the throughput of the same instructions on the Instinct MI355X** , specifically targeting activation and softmax bottlenecks. (The preliminary CDNA 5 ISA Reference Guide does not publish a per-cycle instruction-rate table for transcendental opcodes, though LLVM`gfx1250` scheduling definitions also show FP32 transcendental latency dropping from 9 cycles on RDNA 4 to 8 cycles on`gfx1250` .)\n  - The ISA also adds native **BF16 transcendental instructions** (`V_EXP_BF16` ,`V_LOG_BF16` ,`V_RCP_BF16` ,`V_RSQ_BF16` ,`V_SQRT_BF16` ,`V_SIN_BF16` ,`V_COS_BF16` ), hardware**hyperbolic tangent** (`V_TANH_F32` ,`V_TANH_F16` ,`V_TANH_BF16` , directly accelerating attention logit soft-capping), and single-lane**pseudo-scalar transcendental ALU ops** (`V_S_EXP_F32` ,`V_S_EXP_F16` ,`V_S_LOG_F32` ,`V_S_LOG_F16` ,`V_S_RCP_F32` ,`V_S_RCP_F16` ,`V_S_RSQ_F32` ,`V_S_RSQ_F16` ,`V_S_SQRT_F32` ,`V_S_SQRT_F16` ).\n  - For general vector math, CDNA 5 retains packed 64-bit FP32 operations (`V_PK_FMA_F32` ,`V_PK_ADD_F32` ,`V_PK_MUL_F32` ) and adds RDNA-style**dual-issue VALU instructions** (`VOPD` 64-bit 2-source and a new`VOPD3` 96-bit 3-source encoding), delivering**315 TFLOPS** of peak vector/matrix FP32 and vector FP16 throughput across the GPU.\n4. Just as NVIDIA doubled \n5. **The  Matrix Scaling Leap in FP8/MXFP8 and MXFP4:** Now compare that  transcendental speedup against what CDNA 5 did to the matrix engines per clock:\n  - **Matrix FP16 / BF16 (`V_WMMA_F32_16X16X32_{F16,BF16}`, ):** Doubled  per clock over MI355X to**5.03 PFLOPS** (, or  - **OCP MXFP8, FP8, and MXFP6 (`V_WMMA_F32_16X16X128_{FP8,BF8}` and `V_WMMA_SCALE{,16}_F32_16X16X128_F8F6F4`, ):** Quadrupled **per clock** over MI355X to**20.13 PFLOPS** (, or**FP8/MXFP8 matrix peak () is  the BF16 matrix peak ()** —not —because CDNA 5 expanded the FP8/F6 WMMA reduction depth to  ( BF16’s ).\n  - **OCP MXFP4 (`V_WMMA_SCALE{,16}_F32_32X16X128_F4`, ):** Quadrupled **per clock** over MI355X to**40.26 PFLOPS** (, or\n\nHere is what that -vs.- asymmetry does to the  SIMD tile (, ) when moving from MI355X (`gfx950`) to MI455X (` gfx1250`):\n\n- In **BF16** , both matrix throughput and transcendental throughput doubled () per WGP/CU over MI355X, keeping the BF16 matrix-to-softmax ratio unchanged.\n- In **FP8 / MXFP8** , however, per-WGP matrix throughput scaled- And in **MXFP4** (), both GEMMs for a  tile finish in\n\nThe same squeeze shows up in AMD’s datasheet totals. MI455X lists **315 TFLOPS** of vector FP32 against **20.13 PFLOPS** of dense FP8 matrix, a **64:1** ratio of matrix to vector FLOPs. On NVIDIA’s published numbers, Rubin sits at about 135:1 (§1.2.1). AMD left more vector headroom per matrix FLOP than NVIDIA did, but the software exponential tricks of §2 still run on a vector side that grew more slowly than the matrix side.\n\n#### 1.3.3 What about the rest of the MI400 family (MI450, MI430X, MI440X)?\n\nAMD’s naming moved during 2026, so here is what each name refers to and whether it changes the argument:\n\n- **MI450** is the earlier label for the whole generation. AMD’s Q2 2026 results still say “MI450 Series” for its Helios deals, and this week’s SGLang and vLLM notes say “MI450” for`gfx1250` hardware. At the July 23, 2026 launch AMD defined the MI400 series as two GPUs, MI455X and MI430X.\n- **MI455X** is the Helios rack GPU and the only MI400 part with a full public datasheet. Everything in §1.3.2 refers to it. The datasheet also shows how far CDNA 5 was split by precision: MI455X lists**5 TFLOPS** of FP64, matrix or vector.\n- **MI430X** is the HPC and sovereign-AI part:**288 TFLOPS** of hardware FP64, 432 GB of HBM4, availability in 2027, and systems announced for ORNL’s Discovery and two European machines. AMD has not published its low-precision matrix rates or its transcendental rate. It does not change the attention ledger, and nothing published lets you compute one for it.\n- **MI440X** was announced at CES 2026 as an eight-GPU enterprise server built from a low-precision CDNA 5 subset, like MI455X. It was not in the July launch definition, and AMD has published no compute, memory or availability figures for it.\n\nSo for attention on AMD’s 2026–27 parts, the MI455X numbers above are the only ones there are. The CES framing does support one point: AMD is now building different CDNA 5 dies or configurations per precision envelope (FP64 for MI430X, FP4–BF16 for MI440X and MI455X). That is the same choice NVIDIA made in Rubin’s FP64 vector rate, which fell from 40 TFLOPS on Blackwell to 33. Vendors are trading high-precision units for low-precision matrix area. On the AI parts, exponentials got at most out of that trade (on MI455X, and only for FP16 on Rubin), while FP8 matrix got (MI455X over MI355X) or (Rubin over B300).\n\nPut the cycle budgets across NVIDIA and AMD generations side by side for a standard attention tile (, normalized to a tile on an NVIDIA SM / AMD 4-SIMD WGP, and shown per-SIMD for AMD’s tile):\n\n| Accelerator (Architecture) | Matrix Throughput (BF16 / FP8 / FP4) | Exp / Vector Unit Scaling | BF16 Tile Cycles ( vs. ) | FP8 / MXFP8 Tile Cycles ( vs. ) | FP4 Tile Cycles ( vs. ) | Primary Forward Bottleneck in FP8 / FP4 | \n|---|---|---|---|---|---|---|\n| **NVIDIA Hopper H100** (`sm_90` ) | / / — |  | vs. ( MMA) | vs.  (  **tied** ) | — | **Co-bottlenecked in FP8** ( ratio) | \n| **NVIDIA Blackwell B200** (`sm_100` ) | / / |  **( vs. H100)** | vs.  (  **tied** ) |  **vs.** ( **SFU-bound** ) |  **vs.** ( **SFU-bound** ) | **Severely SFU-bound** ( FP8,  FP4) | \n| **NVIDIA Blackwell Ultra B300** (`sm_103` ) | / / |  **( vs. B200)** | vs. ( MMA) |  **vs.** ( **tied** ) |  **vs.** ( **SFU-bound** ) | **Zero-slack tie in FP8;  SFU-bound in FP4** | \n| **NVIDIA Vera Rubin** (`sm_107` ), FP32`EX2` | / / |  **(FP32)** ( vs. B300) | vs. ( MMA) |  **vs.** ( **SFU-bound** ) |  **vs.** ( **SFU-bound** ) | **Back to B200’s ratios with an FP32 exponential** | \n| **NVIDIA Vera Rubin** (`sm_107` ), FP16`EX2` | same |  **(FP16)** ( vs. B300) | vs. ( MMA) |  **vs.** ( **tied** ) |  **vs.** ( **SFU-bound** ) | **Zero-slack tie in FP8 only if the exponential runs in FP16** | \n| **AMD Instinct MI355X** (`gfx950` , CDNA 4) | /  /   ( `MFMA` ) | Baseline VALU (, per-CU rate not in ZYFA) | vs. ( SIMD tile) |  **vs.** ( **of MFMA** ) | vs. ( VALU-bound) | **VALU softmax + static wave overlap** ( idle in AITER) | \n| **AMD Instinct MI455X** (`gfx1250` , CDNA 5) | /  /   ( `WMMA` ) |  **Transcendentals** vs. MI355X (exact ops/clk unpublished) | vs. ( SIMD tile) |  **vs.** ( matrix vs.  trans) |  **vs.** ( BF16 matrix rate) |  **FP8/FP4 matrix leap outruns  trans scaling** | \n\n**Note: B300 (`sm_103`) supports dense  for NVFP4 ( B200’s ), and Rubin (` sm_107`) expands NVFP4 to  ( B200’s ) while doubling FP8  depth. Rubin’s `EX2` rates are NVIDIA’s published 32 (FP32) and 64 (FP16) ops/clk/SM. For MI455X (`gfx1250`), per-WGP matrix rates are derived directly from AMD’s published peaks ( BF16,  FP8/MXFP8,  MXFP4 at  across 256 WGPs); exact per-cycle transcendental instruction rates are not published in the preliminary CDNA 5 ISA guide, so  brackets MI355X’s 896-cycle exact softmax under the whitepaper’s stated  transcendental throughput:  if all of those cycles were transcendental,  if none were.*\n\n**Conclusion:** Across both NVIDIA (`sm_100`  `sm_107`) and AMD (` gfx950`  `gfx1250`), the transition from BF16 to FP8 and FP4 broke the foundational assumption of early FlashAttention kernels—that non-matmul softmax work is cheap enough to hide effortlessly behind slow matrix multiplications. Once matrix units take 128 to 1,024 cycles per tile while exact softmax takes 256 to 1,024 cycles, hardware SFU/transcendental doublings alone only tread water. They fall further behind when FP8/FP4 matrix depth quadruples (CDNA 5) or when the doubling covers only FP16 exponentials (Rubin). To keep the matrix engines fed, software had to rewrite how exponentials and online normalization are computed.\n\n## 2. How does software get around the exponential unit, and what does it cost?[#](#2-how-does-software-get-around-the-exponential-unit-and-what-does-it-cost)\n\nWhen hardware transcendental units cannot keep up with low-precision matrix engines, kernel engineers have only two mathematical degrees of freedom:\n\n1. **Replace hardware `exp2` calls with software bit-manipulation and polynomial FMA instructions** that execute on underutilized ALU/FMA pipes; and\n2. **Eliminate redundant vector rescaling and reduction operations** in the online softmax loop.\n\nBoth **FlashAttention-4** (on NVIDIA Blackwell B200) and **Zyphra’s ZYFA** (on AMD MI355X) pull these exact levers—and AMD’s **CDNA 5 (`gfx1250`)** ISA bakes several of those software workarounds directly into silicon.\n\n### 2.1 How does FlashAttention-4 do it on Blackwell?[#](#21-how-does-flashattention-4-do-it-on-blackwell)\n\nHere is how **FlashAttention-4** attacks the  `MUFU.EX2` wall on Blackwell B200 (`sm_100`).\n\nFirst, FA4 partitions each 512-thread CTA into four specialized 128-thread warpgroups (`WG0`–` WG3` in the CuTe-DSL / TIRx reference implementation) and uses PTX `setmaxnreg` instructions to dynamically redistribute the SM’s 65,536 32-bit registers to the warps that actually hold the softmax rows:\n\n- **`WG3` (Orchestration: Warp 0 = `tcgen05` MMA, Warp 1 = TMA Load, Warp 2 = TMA Store):** Because`tcgen05.mma` is launched by a single elected thread (`elect_sync` ) and writes its accumulator directly to Tensor Memory (TMEM) rather than registers,`WG3` sheds registers via`setmaxnreg.dec.sync.aligned.u32(48)` (**48 registers/thread** ).\n- **`WG2` (Output Correction & Epilogue):** Sheds registers via`setmaxnreg.dec.sync.aligned.u32(64)` (**64 registers/thread** ).\n- **`WG0` and `WG1` (Ping-Pong Softmax for Q Stages 0 and 1):** Acquire the freed registers via`setmaxnreg.inc.sync.aligned.u32(200)` (**200 registers/thread** , totaling  registers). That 200-register budget allows each of the 128 threads in a softmax warpgroup to load and hold**one entire 128-element FP32 row of** in private registers (`s_chunk_buf` ), eliminating inter-thread warp shuffles during the row-max and row-sum reductions.\n\nEven with zero warp-shuffle overhead and `WG0`/` WG1` alternating on two query tiles ( and ), the hardware `MUFU.EX2` pipe is still capped at . While `MUFU.EX2` is 100% saturated during softmax, the SM’s general-purpose FP32 FMA pipelines sit mostly idle.\n\nFA4 exploits those idle FMA pipelines by implementing a **hybrid hardware/software exponential**:\n\n1. **Cody–Waite Range Reduction ():** For any scaled logit  (clamped at  to prevent underflow), decompose  into its integer floor  and fractional part . In hardware, FA4 extracts  in two FP32 instructions using the magic-number round-down trick: add  to force the fractional bits out of the 23-bit IEEE-754 mantissa, then subtract it back in round-toward- mode.\n2. **Degree-3 Sollya Minimax Polynomial on Packed `f32x2` FMA Units:** On the unit interval , FA4 approximates  with a degree-3 polynomial whose coefficients are optimized via the Sollya package to minimize maximum relative error:\nUsing Horner’s rule, evaluating this cubic polynomial on pairs of logits takes**just three packed `fma.rn.ftz.f32x2` instructions** :\n3. **IEEE-754 Exponent Bit-Shift Reconstruction:** Because , its IEEE-754 FP32 representation already has biased exponent  and holds the fractional mantissa bits in bits . Multiplying by  is therefore a single integer shift-and-add: shift the integer  left by 23 bits into the FP32 exponent field (`[23..30]` ) and add the bit representation of .\n\nWhy doesn’t FA4 run *all* 128 exponentials per row through FMA emulation? Because polynomial emulation requires extra temporary registers for intermediate values and polynomial coefficients, consumes more register-file read/write bandwidth, and has higher instruction latency than a single `MUFU.EX2`. Emulating 100% of the row would spill registers past the 200-register ceiling. Instead, FA4 tunes the split (`emu_pairs`): **75%–90% of each row executes on hardware `MUFU.EX2`, while 10%–25% executes in parallel on the FMA units**, expanding effective exponential throughput just enough to break the `MUFU` bottleneck without a single register spill.\n\nAnd what about numerical accuracy? FA4 measured the error across 4 million random inputs in against an FP64 oracle:\n\n| Exponential Method | FP32 Max Rel. Error | FP32 Mean Rel. Error | **BF16 Max Rel. Error** (after cast to ) | **BF16 Mean Rel. Error** (after cast to ) | \n|---|---|---|---|---|\n| **Ideal ()** | — | — |  |  | \n| **Hardware `MUFU.EX2`** |  |  |  |  | \n| **FA4 Degree-3 FMA Polynomial** |  |  |  |  | \n| **Degree-4 FMA Polynomial** |  |  |  |  | \n| **Degree-5 FMA Polynomial** |  |  |  |  | \n\nAt the raw FP32 level, the 3-FMA cubic polynomial has a max relative error of . Once cast to BF16 (or FP8) to form the operand  for , however, **BF16’s 7-bit mantissa rounding error () is  larger than the polynomial approximation error.** After rounding to BF16, the degree-3 FMA polynomial matches hardware `MUFU.EX2` to within **1 BF16 ULP on 99% of inputs** and achieves the exact same mean relative error ().\n\nFA4 pairs this with a second non-matmul optimization: **Conditional Online Softmax Rescaling**.\nIn standard FlashAttention-2 and -3, whenever KV block  produces a candidate row maximum  strictly greater than the previous reference , the kernel immediately rescales the running output accumulator  by . On Blackwell, where  lives in TMEM (`O_region`), rescaling  is expensive: the Correction warpgroup (` WG2`) must read the  FP32 tile out of TMEM into registers (` tcgen05.ld`), multiply every element by , and write it back to TMEM (` tcgen05.st`) before the next  MMA can accumulate into .\n\nFA4 notices that FP32 has a massive exponent range ( to ), so subtracting instead of does not threaten overflow unless the new maximum jumps by a large margin. Setting a threshold (allowing unnormalized block weights up to ), FA4 modifies the online update rule to:\n\nEach warp in `WG2` checks its 32 rows simultaneously using `vote.sync.any.pred` (`any_needs_rescale`). Whenever all 32 rows stay within the  slack window (), the warp skips the entire `TMEM -> registers -> multiply -> TMEM` correction pass and immediately signals the `p_o_rescale` barrier so the  MMA can proceed. Because the final epilogue normalizes  by the true accumulated denominator  on the same reference scale, skipping intermediate rescales preserves exact mathematical equivalence while **cutting vector rescaling operations by ~10×**. Combined with its CuTe-DSL async pipeline, FA4 reaches **1,613 TFLOP/s** in BF16 on B200 (**71% hardware utilization**,  faster than cuDNN 9.13 and  faster than Triton).\n\n### 2.2 How does ZYFA do it on MI355X (`gfx950`), and what changes on CDNA 5 (` gfx1250`)?[#](#22-how-does-zyfa-do-it-on-mi355x-gfx950-and-what-changes-on-cdna-5-gfx1250)\n\nOn AMD’s Instinct MI355X (`gfx950`), the problem is even sharper: in FP8 attention, the target probability matrix  consumed by the  MFMA instruction is not 16-bit BF16; it is **8-bit FP8 `e4m3`**.\n\nIn AMD’s default AITER kernel, computing  follows the textbook path: evaluate the FP32 base-2 exponential on the vector ALU (`v_exp_f32`), scale it, and convert the FP32 result down to FP8 `e4m3`. Across a  tile, that exact FP8 softmax takes **896 vector cycles** against **1,024 matrix cycles**, starving the matrix pipe.\n\nIn **ZYFA**, Zyphra asked a simpler question: *if the destination of  is an 8-bit `e4m3` float with only 3 mantissa bits, why compute a 23-bit FP32 transcendental on the SFU/VALU only to throw away 20 bits of mantissa a cycle later?*\n\nThe binary layout of an unsigned FP8 `e4m3` number:\n\n- Bit 7 is the sign bit ( for non-negative probabilities ).\n- Bits are the 4-bit biased exponent (bias ).\n- Bits are the 3-bit mantissa , representing the significand .\n\nNow read that 8-bit `e4m3` byte directly as an unsigned integer :\n\nBecause an `e4m3` float has 3 mantissa bits ( quantization steps per octave), **its integer code  grows by exactly 8 every time the floating-point value doubles ().** And on the fractional interval , the piecewise-linear secant  already approximates  to within a maximum error of , which is smaller than a single `e4m3` mantissa step ().\n\nThis observation—a modern FP8 incarnation of Schraudolph’s (1999) IEEE-754 bit-level exponential—gives ZYFA its core mechanism, **Lean P**:\n\n1. **One Multiply-Add + One Integer Conversion:** Instead of calling the hardware exponential unit and then casting FP32 to FP8,**Lean P** computes the integer bit pattern of the`e4m3` probability directly from the scaled logit  using**a single floating-point multiply-add () and a single float-to-integer conversion** . The low byte of that integer*is* the valid`e4m3` floating-point probability.\n2. **57% Reduction in Softmax Cycles:** Eliminating the transcendental instruction and separate float-to-FP8 conversion slashes softmax vector work per  block from**896 vector cycles down to 384 vector cycles** ( fewer VALU cycles).\n3. **Two-Wave Ping-Pong Schedule:** With softmax compressed to 384 vector cycles against 1,024 matrix cycles, ZYFA pairs two waves on each SIMD in a ping-pong schedule: while Wave 0 issues MFMA matrix instructions, Wave 1 computes Lean P probabilities on the VALU, and then they swap. Because 384 vector cycles now fit comfortably inside the 1,024-cycle MFMA window,**steady-state matrix-pipe utilization jumps from 60% (with exact P) to 86% (with Lean P)** .\n4. **Power-of-2 Block Scaling and Rounded Row Sum via MFMA:** To prevent small probabilities from underflowing`e4m3` ’s narrow dynamic range and to eliminate quantization bias in the denominator, ZYFA scales each 64-key block of probabilities by a power of 2 before the Lean P rounding step—a factor that CDNA 4’s block-scaled MFMA instruction removes in hardware at**zero extra cycle cost** —and normalizes by the sum of the*rounded* FP8 probabilities (computed by issuing**1 extra matrix instruction per block** rather than burning VALU cycles on a vector reduction).\n\nWhat **Lean P + block scaling + rounded row sum** achieves on MI355X:\n\n- **Throughput:** On 32K-token Gaussian benchmarks, ZYFA reaches**2,672 TFLOP/s** in FP8 (**53% of MI355X’s FP8 datasheet peak** , vs. AITER’s**1,944 TFLOP/s / 39%** —a**1.37×** gain) at , and on DeepSeek’s MLA shape () it reaches**2,464 TFLOP/s vs. AITER’s 1,237 TFLOP/s (1.99×)** .\n- **Accuracy (62%–78% Lower Error than Exact Hardware Exp Without Block Scaling):** Against an FP64 reference, even though Lean P uses a 1-FMA bit-pattern approximation for , ZYFA’s power-of-2 block scaling and rounded row sum cut total FP8 attention error to **AITER’s on Mistral-7B (78% lower error)** and **on Qwen3-8B (62% lower error)** . Across the full Mistral-7B model, KL divergence from FP64 drops from  (AITER) to **(ZYFA)** , with bitwise-identical outputs across 7,200 consecutive kernel launches.\n\n#### How does CDNA 5 (`gfx1250`, MI455X) change the AITER / ZYFA pipeline?\n\nWhen porting AITER, ZYFA, or FlyDSL attention kernels from MI355X (`gfx950`) to **CDNA 5 (` gfx1250`, MI450 / MI455X)**—where initial `gfx1250` enablement for `gpt-oss-120b`, `DeepSeek-R1`, `DeepSeek-V4-Flash`, `MiniMax-M3`, and FlyDSL / AITER `tgemm` landed in ROCm 10.0 / 10.1, vLLM v0.33.0 (`#58761, #58800`), and SGLang v0.5.23 (`#41988, #42699, #44671, #45304, #45365, #45758, #46366`) this week—six ISA changes directly alter the attention cycle ledger:\n\n1. **4× Addressable Unified VGPRs (1,024 VGPRs/Wave via `S_SET_VGPR_MSB`):** On CDNA 4, a wave was restricted to 256 architectural VGPRs plus 256 MFMA-only`AccVGPRs` . On CDNA 5, the split`AccVGPR` file is eliminated and a single Wave32 can allocate up to**1,024 unified 32-bit VGPRs** (in blocks of 16). To address registers beyond`V255` without breaking 8-bit register fields in standard encodings, CDNA 5 introduces`S_SET_VGPR_MSB` (and`MODE` bits`DST_VGPR_MSB` ,`SRC0_VGPR_MSB` ,`SRC1_VGPR_MSB` ,`SRC2_VGPR_MSB` ), which supplies the upper 2 bits (`[9:8]` ) of each operand’s 10-bit VGPR address. (While RDNA 4’s dynamic VGPR allocation instruction is a no-op on`gfx1250` , static allocation of up to 1,024 VGPRs per wave allows an attention wave to keep large , , and  subtiles resident in registers without`AccVGPR` -to-`VGPR` copies or LDS spills.)\n2. **Hardware VALU/WMMA Co-Execution, `S_DELAY_ALU`, and 16 Named Barriers per Workgroup:** On MI355X, overlapping VALU softmax with MFMA required brittle static instruction-order interleaving across waves. On CDNA 5, 16-bit and smaller XDL`WMMA` instructions are tracked in`S_DELAY_ALU` as`TRANS` operations (`DEP` codes 5–7), allowing normal VALU instructions and`WMMA` instructions to**co-execute** . By default, the arbiter stalls a wave for the multi-cycle duration of its`WMMA` so peer waves on the same SIMD can co-execute VALU softmax instructions; setting`SCHED_MODE[2]` (`DISABLE_XDL_ARB_STALL` ) instead allows a single wave to issue multiple`WMMA` instructions back-to-back and immediately issue independent VALU instructions while the matrix ops execute. Combined with**16 hardware named barriers per workgroup** (`S_BARRIER_INIT` ,`S_BARRIER_JOIN` ,`S_BARRIER_SIGNAL` ,`S_BARRIER_WAIT` , allocated from 64 per WGP) and**LDS split barriers** (`DS_ATOMIC_ASYNC_BARRIER_ARRIVE_B64` ), CDNA 5 supports true producer/consumer wave specialization.\n3. **Dual-Issue `VOPD` / `VOPD3` and Packed `V_PK_FMA_F32`:** Because MI455X quadruples FP8/MXFP8 and MXFP4 matrix throughput () while doubling transcendental throughput (), software exponential paths like ZYFA’s**Lean P** (1 FMA + 1 int cast) or FA4’s**Horner cubic polynomial** become even more essential on CDNA 5. CDNA 5 accelerates those non-transcendental softmax instructions with both 64-bit packed FP32 math (`V_PK_FMA_F32` ,`V_PK_MUL_F32` ,`V_PK_ADD_F32` ) and**dual-issue VALU encodings** (`VOPD` and the new 96-bit 3-source`VOPD3` , which removes`VOPD` ’s even/odd destination bank restriction).\n4. **Pseudo-Scalar Transcendentals (`V_S_EXP_F32`, `V_S_RCP_F32`) and Native `V_TANH_*` / `V_EXP_BF16`:** In Wave32 attention, when a row or sub-tile maximum  and denominator  are reduced across lanes, computing the per-row rescaling factor  or final reciprocal  previously burned a full 32-lane vector transcendental instruction. CDNA 5’s**pseudo-scalar transcendental ops** (`V_S_EXP_F32` ,`V_S_EXP_F16` ,`V_S_RCP_F32` ,`V_S_RCP_F16` ) read an SGPR/constant and write the result directly to an SGPR in a single lane, while`V_TANH_{F32,F16,BF16}` and`V_EXP_{F16,BF16}` accelerate half-precision activation and soft-capping steps.\n5. **Hardware Block-Scaled Conversions (`V_CVT_SCALEF32_PK*`) + 4-DWORD Block-Scaled `V_WMMA_SCALE*`:** ZYFA’s biggest accuracy win on MI355X came from computing a per-block power-of-2 scale in software before casting  to FP8. CDNA 5 turns block-scaled conversion into a native VALU instruction family:`V_CVT_SCALEF32_PK8_{FP8,BF8,FP4}_{F32,F16,BF16}` and`V_CVT_SCALEF32_PK16_{FP6,BF6}_{F32,F16,BF16}` (plus stochastic-rounding variants`V_CVT_SCALEF32_SR_*` ), which feed directly into CDNA 5’s 4-DWORD`VOP3PX2` block-scaled matrix instructions (`V_WMMA_SCALE_F32_16X16X128_F8F6F4` for 32-element blocks and`V_WMMA_SCALE16_F32_16X16X128_F8F6F4` for 16-element blocks, supporting`E8M0` ,`E5M3` , and fractional`E4M3` scales).\n6. **Tensor Data Mover (TDM), Async Global-to-LDS Loads, Cluster Multicast, and Transpose Loads (`DS_LOAD_TR*` / `GLOBAL_LOAD_TR*`):** Each WGP on MI455X includes a dedicated**Tensor Data Mover (TDM)** (`TENSOR_LOAD_TO_LDS` ,`TENSOR_STORE_FROM_LDS` , tracked by`TENSORcnt` ) that copies up to 5D tiles between HBM4 and the 320 KB LDS without using VGPRs, optionally inserting bank-conflict-free LDS padding (`pad_enable` ), broadcasting tiles across up to 16 WGPs in a Shader Engine cluster (`workgroup_mask` ), or executing indexed row gathers (up to 16 16-bit or 8 32-bit row indices per descriptor). Once in LDS (or directly from global memory),**`DS_LOAD_TR16_B128`, `DS_LOAD_TR8_B64`, `DS_LOAD_TR6_B96`, and `DS_LOAD_TR4_B64`** (and` GLOBAL_LOAD_TR*` ) load and transpose 16-bit, 8-bit, 6-bit, or 4-bit matrix tiles into VGPRs in hardware, eliminating the VALU/LDS transpose instructions that previously competed with softmax.\n\n### 2.3 Why did the faster kernel run at a lower clock on MI355X?[#](#23-why-did-the-faster-kernel-run-at-a-lower-clock-on-mi355x)\n\nWhen Zyphra measured ZYFA and AITER on **captured real-model activations at sustained clocks** (16K context on Mistral-7B, Qwen3-8B, and DeepSeek-V2-Lite), they uncovered one of the most important hardware-software interactions in modern low-precision inference.\n\nWhile ZYFA achieved a **1.99×** speedup on synthetic Gaussian inputs (and **1.76×** on DeepSeek-V2-Lite’s real MLA activations), its wall-clock speedup on real Mistral-7B and Qwen3-8B activations was **1.16× and 1.17×**—significantly smaller than the cycle-count reduction predicted. Why?\n\nBecause on the Mistral-7B layer, **AITER ran at a sustained GPU clock of 2,108 MHz, while ZYFA caused the MI355X to throttle its clock down to 1,628 MHz—a 480 MHz (22.8%) frequency drop.**\n\nWhy did a faster, more accurate kernel run at a 22.8% lower clock frequency on the exact same GPU and the exact same layer activations?\n\nThe answer is dynamic CMOS power in the multiplier array:\n\nwhere  is the **bit-toggling activity factor**—the fraction of logic gates and operand bits that flip between  and  on each clock cycle.\n\n1. **Why AITER runs at 2,108 MHz:** In real language-model attention (unlike synthetic zero-mean Gaussian tensors), attention distributions are heavy-tailed: a few keys have large probabilities, and thousands of keys in the tail have very small probabilities. Without ZYFA’s per-64-key power-of-2 block scaling, AITER lets those small tail probabilities**underflow to exact zero (`0x00`) or subnormals** when cast to FP8` e4m3` . When the  MFMA array streams thousands of`0x00` bytes into its multiplier trees, the operand bits do not flip (). Dynamic switching power drops, and MI355X’s power management firmware boosts  to**2,108 MHz** .\n2. **Why ZYFA drops to 1,628 MHz (and still wins):** ZYFA’s per-64-key power-of-2 block scaling rescales each block upward before rounding so that small tail probabilities survive as**non-zero `e4m3` bit patterns** (which is why ZYFA’s error is 62%–78% lower). Feeding dense, non-zero mantissa and exponent bits into the Matrix Cores on every cycle maximizes bit-toggling activity  at 86% matrix-pipe utilization, driving the chip into its TDP/current limit and forcing DVFS to pull the clock down to**1,628 MHz** . As Zyphra’s authors emphasize, this is why low-precision attention kernels must always be benchmarked on real model activations at sustained clocks rather than synthetic Gaussian inputs in short bursts.\n\nEven after absorbing that 22.8% clock drop, ZYFA’s cycle reduction is so large that it still finishes **1.16×–1.76× faster in wall-clock time** and delivers **21%–22% higher energy efficiency** (**1.94 and 1.91 TFLOP/J** on Mistral-7B and Qwen3-8B, vs. AITER’s **1.60 and 1.56 TFLOP/J**).\n\n| Kernel & Target GPU | Softmax Mechanism | Softmax Cycles / Tile | Matrix Pipe Steady-State Busy % | Headline FP8 / BF16 Throughput | Real-Activation Sustained Clock (Mistral-7B) | Energy Efficiency & Numerical Error vs. FP64 | \n|---|---|---|---|---|---|---|\n| **AMD AITER FP8** (MI355X,`gfx950` ) | Hardware `v_exp_f32` + unscaled FP8 cast | () | (ping-pong) (default) | () (MLA) |  **(tail probs underflow to** `0x00` ) | ; baseline FP8 error () | \n| **Zyphra ZYFA FP8** (MI355X,`gfx950` ) | **Lean P** (1 FMA + 1 int cast) +  block scale + MFMA row sum |  **()** |  |  **()** **(MLA, )** |  **(non-zero bits toggle multipliers)** | **;**  **lower error** () | \n| **FlashAttention-4 BF16** (Blackwell B200,`sm_100` ) | **Dual-path:** 75–90%`MUFU.EX2` + 10–25%**Degree-3 FMA** + cond. rescale () | effective |  **of GPU peak** |  **BF16 ( cuDNN 9.13,  Triton)** | Sustained 1,000W TDP | Matches `MUFU.EX2` within  on  of inputs | \n\n**Conclusion:** On both Blackwell and MI355X (and even more so under CDNA 5’s  low-bit matrix scaling), the only way software reaches 53%–71%+ of peak attention throughput is by treating the non-matmul softmax path—every `MUFU.EX2` instruction, every VALU cycle, every TMEM load/store, and every bit-flip in the probability tile—as the primary scarce resource. Now watch what happens when a hardware architect tries to bolt **2:4 Sparse Tensor Cores** onto that exact bottleneck.\n\n## 3. Can 2:4 sparse Tensor Cores speed up attention?[#](#3-can-24-sparse-tensor-cores-speed-up-attention)\n\nOnce you understand that modern attention is bottlenecked by softmax and the on-chip data path between  and , examine the hardware pitch for **2:4 Sparse Tensor Core Attention**: most recently in NVIDIA’s **Vera Rubin (`sm_107`)** disclosures and PTX ISA 9.4, and in AMD’s 4:2 structured-sparse `V_SWMMAC` instructions on **CDNA 5 (`gfx1250`)**.\n\nHere is the hardware pitch in its strongest form:\n\n1. **In NVIDIA’s Vera Rubin (`sm_107`) architecture preview and PTX ISA 9.4:** The first attention GEMM () runs densely on Tensor Cores, writing the dense FP32 score tile  into Tensor Memory (TMEM). Next, instead of loading all  scores into registers with a standard`tcgen05.ld` , Rubin introduces a new compound PTX 9.4 instruction:\nWhile reading the dense FP32 scores out of TMEM,`tcgen05.ld.red.spcompress` performs**2:4 selection in hardware** : from every group of 4 consecutive FP32 logits, it retains the 2 largest values (`cdata` ), generates 2-bit-per-element positional metadata (`mdata` ), and simultaneously computes the fragment’s local row maximum (`redval` via`.red` with`rowop::max` ).\nBecause only 50% of the logits survive into`cdata` , the softmax warps only have to evaluate`ex2.approx((x - max) * log2(e))` and normalize on**half as many values** (cutting SFU work in half), and the second attention GEMM () can then execute on**Sparse Tensor Cores (`tcgen05.mma.sp`)** at up to  the dense MMA instruction rate.\n2. **The Broader 2:4 Hardware Promise:** Since Ampere (`sm_80` ) in 2020, NVIDIA’s keynote slides and datasheets have quoted “2× the math throughput of dense matrix units” whenever an operand satisfies the 2:4 pattern (at least 2 zeros in every contiguous 4-element group along the GEMM reduction dimension , encoded with 2-bit positional indices).\n\nEven on **static linear weights**—where pruning and metadata compression happen offline before deployment—practitioners have learned to treat the “2× throughput” datasheet figure with skepticism. Across NVIDIA’s own benchmarks, SemiAnalysis’s Hopper Tensor Core analysis and Spheron’s production benchmarks, real-world 2:4 sparse GEMMs on static weights typically land between  **and**  over dense equivalents (e.g.,  on A100 TensorRT 8.0,  single-stream on A100 Sparse Llama, and **—zero measured speedup—on an NVIDIA L4 serving a verified 2:4-pruned Qwen3-4B-Instruct across concurrency 1–128**), due to sparse kernel maturity gaps relative to `cuBLAS`, TDP power ceilings, and memory-bandwidth bottlenecks at low batch sizes. SemiAnalysis put it bluntly in its Rubin CPX analysis: sparsity “has yet to actually deliver the promised benefits – falling well short of the 2x pickup it promises.”\n\nThe 2026–27 datasheets still lead with sparse numbers, and the two vendors now do it differently:\n\n- **NVIDIA Rubin introduces a third sparsity scheme.** NVIDIA’s Rubin table gives**50 PF NVFP4 “inference”** , footnoted as “Transformer Engine compute,” next to**35 PF NVFP4 “training”** , footnoted as “dense compute.” NVIDIA attributes the gap to “new hardware-accelerated adaptive compression designed to boost NVFP4 performance while preserving accuracy.” SemiAnalysis reads the same ratio as sparse to dense (it puts Rubin at 50 PF sparse vs 33.3 PF dense, and CPX at 30 vs 20, a 3:2 ratio) and says the scheme “is unlike the 2:4 structured sparsity used in Hopper and Ampere, and it isn’t like Blackwell’s 4:8 pairwise structured sparsity either.” NVIDIA has not published what pattern it accepts, whether the compression happens offline on weights or online on activations, or whether it applies to the attention GEMMs at all. Until it does, the honest reading is that Rubin’s dense NVFP4 rate is 35 PF, and the rest of this article’s tests apply as soon as the mechanism is public. If it compresses static weights offline, §4’s equal-bit and capability arguments apply. If it compresses activations such as  online, §3.3’s preprocessing cost and §3.4’s Amdahl ceiling apply.\n- **AMD publishes 2:4 peaks only where they cannot beat its own dense FP8.** The MI455X datasheet lists “with structured sparsity” peaks for exactly three formats, Matrix FP16, BF16 and INT8, each**10,066 TFLOPS** , double their 5,033 dense. It lists no sparse figure for FP8, MXFP8, MXFP6 or MXFP4. On the same chip,**dense FP8 is 20,133 TFLOPS** . So the fastest sparse number AMD will put on paper for MI455X is half of its dense FP8 number. AMD’s MI400 page describes the feature the same way: “INT8, and FP16/BF16 sparsity support for AI.” The ISA does define FP8`V_SWMMAC` instructions (§3.2), but AMD has not published a rate for them.\n\nWhen you move from static weights to **dynamic attention**, where the sparse operand  (or ) must be pruned, normalized, quantized, and packed into hardware metadata format *on the fly* inside every attention tile, the hardware pitch collides with four independent mechanisms.\n\n### 3.1 Where do the operands live on `tcgen05`, and why does that force a 7-step repack?[#](#31-where-do-the-operands-live-on-tcgen05-and-why-does-that-force-a-7-step-repack)\n\nFirst, trace where Tensor Core and Matrix Core operands actually live across NVIDIA and AMD architectures:\n\n| Accelerator Architecture | Matrix Instruction | Operand A ( or ) Location | Operand B ( or ) Location | Accumulator D ( or ) Location | Sparse Metadata Location | \n|---|---|---|---|---|---|\n| **NVIDIA Volta / Ampere / Ada** (`sm_70` –`sm_89` , plus`sm_120` ) | `mma.sync` /`mma.sp.sync` | **Registers (RMEM)** | **Registers (RMEM)** | **Registers (RMEM)** | **Registers (RMEM)** | \n| **NVIDIA Hopper** (`sm_90a` ) | `wgmma.mma_async` /`wgmma.mma_sp` | **RMEM or SMEM** | **SMEM** | **Registers (RMEM)** | **Registers (RMEM)** | \n| **NVIDIA Blackwell & Vera Rubin** (`sm_100a` ,`sm_103a` ,`sm_107a` ) | `tcgen05.mma` /`tcgen05.mma.sp` | **TMEM or SMEM***(never RMEM)* | **SMEM***(never RMEM)* | **TMEM***(never RMEM)* | **TMEM***(never RMEM)* | \n| **AMD CDNA 5** (`gfx1250` , MI455X) | `V_WMMA` /`V_SWMMAC` | **Registers (VGPRs)** | **Registers (VGPRs)** | **Registers (VGPRs)** | **Registers (VGPR `SRC2`)** | \n\nOn older register-resident NVIDIA architectures like Ada Lovelace (`sm_89`), `mma.sp.sync.aligned.m16n8k32` is a warp-synchronous instruction whose inputs , , metadata , and accumulator  **all live in the thread register file (RMEM)**. That register-resident symmetry is what academic Ada kernels such as *DynSpAttn* relied on: when  finishes in thread registers, each thread can prune 2 of 4 scores in-register, run `exp2` in-register, pack two FP16 values and 2-bit indices into registers, and pass those exact registers directly into the next `mma.sp.sync` instruction for  without writing  or its metadata to shared memory.\n\n**That register-resident data path no longer exists on datacenter Blackwell (`sm_100`/` sm_103`) or Vera Rubin (` sm_107`).**\nTo scale tile sizes up to  and  without exhausting the 65,536-register file, `tcgen05` completely severed the Tensor Core from the CUDA Core register file:\n\n- `tcgen05.mma` writes the dense score accumulator  into**Tensor Memory (TMEM)** .\n- CUDA Cores (and SFUs) **cannot operate on TMEM directly** ; softmax warps must load  out of TMEM into general-purpose thread registers (RMEM) via`tcgen05.ld` .\n- Conversely, **`tcgen05.mma` and `tcgen05.mma.sp` cannot read operand A () or sparse metadata from general-purpose thread registers.** Operand A must reside in**TMEM or Shared Memory (SMEM)** , and for`tcgen05.mma.sp` , the 2-bit-per-element metadata operand must also reside at a**TMEM address** .\n\nNow trace what actually happens on Vera Rubin (`sm_107`) when you use PTX 9.4’s `tcgen05.ld.red.spcompress` instruction:\n\n- `tcgen05.ld.red.spcompress` always operates on**FP32 (`f32`)** data in TMEM, performs 2:4 selection, emits 2-bit (` b2` ) metadata, and supports vector lengths from`.x4` to`.x128` .\n- **Where does it put the output?** It places the compressed FP32 logits (`cdata` ) and the 2-bit metadata (`mdata` ) into**general-purpose thread registers (RMEM)** .\n- Even inside the register file, `ld.spcompress` does not cut register pressure by 50%: metadata registers add overhead for shorter vector widths, while long vector widths (`.x128` ) inflate peak register live ranges inside the exact softmax warpgroups (`WG0` /`WG1` ) that are already pinned against their 200-register/thread ceiling.\n- Furthermore, `rowop::max` only returns the local maximum`redval` of the loaded fragment—it is**not** a complete softmax instruction.\n\nBecause `cdata` and `mdata` land in thread registers while `tcgen05.mma.sp` requires sparse operand A in **SMEM or TMEM** and its metadata in **TMEM**, the software pipeline between  and  is forced to execute a **7-step register-to-TMEM/SMEM repack round-trip** on every single tile:\n\nCompare that 7-step sparse obstacle course against what **dense FlashAttention-4** does on the same TMEM hardware:\n\n- In dense FA4, the 128-Row  512-Column TMEM allocation (`tmem` ) has**zero metadata region** .`S0` occupies physical columns`[0, 128)` , and`P0` is simply an in-place`float16` view (`P_region` ) aliasing the upper 64 physical columns`[64, 128)` of`S0` .\n- The softmax warpgroup loads , computes , casts to 16-bit/8-bit, and issues contiguous `tcgen05.st.sync.aligned.32x32b.x16.b32` stores straight into`[64, 128)` —immediately kicking off the first segment (`K_SPLIT = 96` ) of the dense  MMA after just three 32-column chunks are written, with**zero metadata repacking, zero metadata TMEM stores, and zero TMEM column fragmentation** .\n\nIn short, on `tcgen05`, `ld.spcompress` saves half the `ex2.approx` calls in Step 1 only to pile Steps 3, 4, and 5 (register layout shuffling, metadata bit-packing, and extra TMEM metadata writes) right back onto the softmax warpgroups’ CUDA Core ALU and TMEM-store pipelines.\n\n### 3.2 Does AMD CDNA 5 (`gfx1250`) avoid that round-trip?[#](#32-does-amd-cdna-5-gfx1250-avoid-that-round-trip)\n\nThe fourth row of the operand table above is **AMD’s CDNA 5 (`gfx1250`, Instinct MI455X)**.\n\nUnlike NVIDIA’s `tcgen05`, AMD did not move matrix accumulators into a separate Tensor Memory address space on CDNA 5. Instead, CDNA 5 expanded each Wave32’s addressable vector register file to **1,024 unified VGPRs** and kept all matrix operands in VGPRs. For structured sparsity, CDNA 5 provides the **`V_SWMMAC`** instruction family:\n\n- `V_SWMMAC_F32_16X16X64_{F16,BF16}` ,`V_SWMMAC_F16_16X16X64_F16` ,`V_SWMMAC_BF16_16X16X64_BF16` , and`V_SWMMAC_BF16F32_16X16X64_BF16` for 16-bit data (where a packed  Matrix A in 8 VGPRs represents a 4:2 sparse  matrix multiplied by a dense  Matrix B in 16 VGPRs);\n- `V_SWMMAC_{F32,F16}_16X16X128_{FP8,BF8}_{FP8,BF8}` and`V_SWMMAC_I32_16X16X128_IU8` for 8-bit data (where a packed  Matrix A in 8 VGPRs represents a 4:2 sparse  matrix multiplied by a dense  Matrix B in 16 VGPRs).\n- In every `V_SWMMAC` instruction, the packed A-matrix (`SRC0` ), dense B-matrix (`SRC1` ),**2-bit sparse index data (`SRC2`, fitting in 1 VGPR where each lane holds sixteen 2-bit codes with `idx0 < idx1`)** , and accumulator C/D (` VDST` )**all reside in VGPRs** .\n\n**Because CDNA 5 keeps `V_WMMA` and `V_SWMMAC` operands and 2-bit index metadata in VGPRs, the `tcgen05` register-to-TMEM/SMEM repack round-trip (§3.1) does not apply to AMD’s sparse path on CDNA 5.** On `gfx1250`, the dense  output of `V_WMMA` is already in VGPRs, and `V_SWMMAC` can consume the pruned  and its index register `SRC2` directly from VGPRs without writing them to LDS or TMEM.\n\n**Does that rescue 4:2 sparse attention on CDNA 5? No—because three CDNA 5 hardware constraints and three general algorithmic walls immediately take its place:**\n\n1. **CDNA 5 has no hardware `spcompress` instruction—so 4:2 selection and index packing run on the VALU:** Unlike Vera Rubin’s`tcgen05.ld.red.spcompress` , the CDNA 5 ISA has no hardware instruction that selects the top 2 of 4 FP32 logits and emits 2-bit indices. On`gfx1250` , software must compare every 4 consecutive FP32 scores in VGPRs, select the top 2 while enforcing the hardware’s ordering rule`idx0 < idx1` , pack the surviving values into`SRC0` , and bit-pack sixteen 2-bit index nibbles per lane into the`SRC2` VGPR—all using VALU instructions on the exact vector pipeline that softmax is already competing for.\n2. **In unscaled FP8, CDNA 5’s dense `V_WMMA` already matches `V_SWMMAC`’s  tile shape:** The ISA’s instruction and data-hazard tables show it. For 16-bit F16/BF16,`V_SWMMAC` doubles the  dimension per instruction from  (`V_WMMA_F32_16X16X32_BF16` , requiring 5`V_NOP` / co-exec hazard slots) to  (`V_SWMMAC_F32_16X16X64_BF16` , 3`V_NOP` slots). For 8-bit FP8/BF8, however, CDNA 5**already provides a dense  instruction (`V_WMMA_F32_16X16X128_FP8_FP8`, 4 `V_NOP` slots)** alongside the  instruction (`V_WMMA_F32_16X16X64_FP8_FP8` , 2`V_NOP` slots), while sparse`V_SWMMAC_F32_16X16X128_FP8_FP8` (3`V_NOP` slots) is capped at the same  shape, so its  matches the wide dense instruction (the`gfx1250` LLVM target definitions show the same). The preliminary CDNA 5 ISA guide publishes no rate for`V_SWMMAC` . The MI455X datasheet gives one only for the 16-bit and INT8 forms: 10,066 TFLOPS, against 20,133 TFLOPS for dense FP8`V_WMMA` .\n3. **CDNA 5’s block-scaled `F8F6F4` and `F4` WMMA instructions do not support structured sparsity at all:** The ISA says of`V_WMMA_SCALE_F32_16X16X128_F8F6F4` ,`V_WMMA_SCALE16_F32_16X16X128_F8F6F4` ,`V_WMMA_SCALE_F32_32X16X128_F4` , and`V_WMMA_SCALE16_F32_32X16X128_F4` :*“These matrix operations do not supported structured sparsity.”* Those block-scaled instructions are the exact instructions that deliver MI455X’s headline**20.13 PFLOPS MXFP8** and**40.26 PFLOPS MXFP4** throughput and implement ZYFA-style hardware block scaling (eliminating 62%–78% of FP8 attention quantization error). On CDNA 5, choosing 4:2`V_SWMMAC` means forfeiting both MXFP4 () and hardware block scaling on FP8.\n\nAnd beyond those ISA specifics, CDNA 5 faces the exact same online pruning overhead (§3.3), single-sided Amdahl ceiling (§3.4), and equal-bit / equal-tier precision and capability losses (§4) as NVIDIA GPUs.\n\n### 3.3 How much does pruning and packing P cost in practice?[#](#33-how-much-does-pruning-and-packing-p-cost-in-practice)\n\nHow much does online pruning, repacking, and metadata compression of the attention probability matrix actually cost in wall-clock time on real GPUs?\n\nIn *Is 2:4 Sparsity Still Worth It?* (October 2026), Xiuyi Qin measured this exact operation across three GPU generations—Ada Lovelace (**L40S**), Hopper (** H100**), and Blackwell (** RTX PRO 6000**, `sm_120`)—using CUPTI hardware kernel timers.\n\nTo give the 2:4 side the fastest possible online compression path, Qin did not rely on `cuSPARSELt`’s built-in `cusparseLtSpMMAPrune` + `cusparseLtSpMMACompress` API (which takes **15,086 µs** on Blackwell and **47,354 µs** on H100 for a   matrix) or PyTorch’s semi-structured conversion (**3,330 µs** on Blackwell, **4,796 µs** on H100). Instead, Qin used a **hand-written fused Triton prune-and-compress kernel** (reimplementing HieraSparse’s kernel bit-for-bit, plus fused FP8 quantization) that reads  once, selects the top 2 of every 4 entries, and writes the compressed values and 2-bit metadata indices once—running at **63% to 102% of the GPU’s raw memory-copy bandwidth** on H100 and Blackwell (**1,892 to 2,277 µs** on Blackwell, **1,530 to 1,728 µs** on H100).\n\nHere is what happens to the attention GEMMs (four query heads per KV group, , sequence lengths  **/** ) when you compare the **standalone sparse GEMM kernel speedup** (ignoring online pruning/compression) against the **true wall-clock speedup once online pruning and compression are included**:\n\n| GPU Architecture | Attention Operand | Same-Precision **FP16** Speed Ratio*(Kernel Only  **With Online Prep**)* | Same-Precision **FP8** Speed Ratio*(Kernel Only  **With Online Prep**)* |  **Tier** vs. Dense FP8*(Kernel  **With Prep**)* |  **Tier** vs. Dense NVFP4*(Kernel  **With Prep**)* | **Share of 2:4 Time Spent in Online Prep** | \n|---|---|---|---|---|---|---|\n| **NVIDIA L40S** (Ada,`sm_89` ) | () | () () | () () | () () | — |  **()  ()** | \n| **NVIDIA H100** (Hopper,`sm_90` ) | () | () () | () () | () () | — |  **()  ()** | \n| **NVIDIA Blackwell** (`sm_120` ) | () | () () | () () | () () | () () |  **()  ()** | \n| **NVIDIA Blackwell** (`sm_120` ) | () | () () | () () | () () | () () | () () | \n| **NVIDIA Blackwell** (`sm_120` ) | () | () () | () () | () () | () () | () () | \n| **NVIDIA Blackwell** (`sm_120` ) | () | () () | () () | () () | () () | () () | \n\nIn the rows:\n\n- When you benchmark the  sparse GEMM kernel in isolation, 2:4 looks like a **1.49× to 1.77×** winner in FP16 (excluding L40S’s narrow-shape dense heuristic outlier at 4K) and a**1.28× to 1.63×** winner in FP8.\n- The moment you include the online pruning and metadata compression of , **online preprocessing consumes 70% to 87% of total runtime.** Why? Because for , the output dimension is the small head dimension . Pruning and compressing an  probability matrix touches  elements, whereas the  GEMM itself only performs  FLOPs per element of . With an arithmetic intensity bounded by , the time spent pruning and packing  is**three to six times larger** than the time saved inside the  Tensor Core instruction.\n- As a result, in FP16, 2:4 on  runs at **0.38× to 0.51× the speed of dense FP16** (more than a  slowdown). And in the  compute tier on Blackwell (comparing 2:4-FP8  against dense NVFP4 ), 2:4 is**slower than dense NVFP4** (**0.93× / 0.96×** ).\n\n### 3.4 How much can pruning only one of the two matmuls save?[#](#34-how-much-can-pruning-only-one-of-the-two-matmuls-save)\n\nThere is an even simpler arithmetic flaw in Rubin’s `ld.spcompress` attention path.\n\nIn Rubin’s described pipeline,  **is computed 100% densely**. Sparse selection only happens *after* , when `tcgen05.ld.red.spcompress` loads  from TMEM to feed .\n\nApply Amdahl’s Law to the two attention GEMMs ( and ), which have identical FLOP counts ( each):\n\n- Even if `tcgen05.mma.sp` (or`V_SWMMAC` ) achieved a realistic  speedup on*and* the register-to-TMEM repack and online pruning cost**zero cycles** , leaving  dense caps the total GEMM speedup at:\n\nTo break past that  single-sided Amdahl ceiling, a kernel has to prune **both** GEMMs—applying 2:4 sparsity to the Query matrix  or Key matrix  along the head dimension  so that  also runs on Sparse Tensor Cores.\n\nThe and rows of the table above show what happens when you apply 2:4 sparsity to or along a 128-channel head dimension:\n\n- Because the reduction axis of  is only , the sparse GEMM kernel has almost no reduction depth over which to amortize metadata fetches: on H100, 2:4 on  runs at - Moreover, any benchmark that claims a  speedup by comparing **2:4-sparse FP16** against**dense FP16** is comparing the wrong baselines. On Ada, Hopper, Blackwell, Rubin, MI355X, and MI455X, the equal-tier dense competitor to 2:4-FP16 is**dense FP8** (such as FA3/FA4 FP8 or ZYFA FP8)—which achieves the same  Tensor Core throughput tier**without** pruning. On MI455X the comparison is not even a tie: AMD’s datasheet puts 2:4 BF16 at 10,066 TFLOPS and dense FP8 at 20,133 TFLOPS, so dense FP8 has twice the peak rate of the sparse 16-bit path, again**without** pruning a single element of  or ,**without** online metadata packing, and with an order of magnitude lower numerical error.\n\n## 4. At equal bits, is 2:4 sparsity better than quantization?[#](#4-at-equal-bits-is-24-sparsity-better-than-quantization)\n\nWhy did **vLLM** (which removed Marlin 2:4 in January 2026 and its Sparse24 compressed-tensors integration in v0.19.0 in March) and **LLM Compressor v0.11.0** delete their 2:4 semi-structured sparsity support in 2026, even as NVIDIA kept Sparse Tensor Cores in Ampere, Hopper, Blackwell, and Rubin?\n\nBecause in real LLM deployment, 2:4 sparsity does not compete against uncompressed 16-bit floats. Because 2:4 composes with quantization, the decision an engineer actually faces at any point on the compression curve is:\n\n**Starting from a given precision, is it better to add 2:4 sparsity, or to drop one step to the next lower dense precision?**\n\nAcross the literature, from analytical rate-distortion and convex SDP bounds to equal-bit and equal-tier evaluations across all six Transformer operands, downstream fact-recall and reasoning benchmarks, and post-2:4 workarounds (*V:N:M* and *SlideSparse*)—quantization dominates 2:4 sparsity everywhere above the hardware precision floor.\n\n### 4.1 How many bits does 2:4 really save, and how much error does it add?[#](#41-how-many-bits-does-24-really-save-and-how-much-error-does-it-add)\n\nBefore even imposing the 2:4 pattern or counting index metadata, ask a more fundamental question: *at equal nominal compression ratios, how does idealized unstructured magnitude pruning compare against uniform quantization?*\n\nIn *Pruning vs Quantization: Which is Better?* (NeurIPS 2023), Kuzmin et al. answered this question analytically, via per-layer optimization bounds, and via full-model fine-tuning—granting unstructured pruning the idealistic assumption of **zero metadata storage overhead** (comparing 50% sparsity directly against INT8, 75% sparsity against INT4, and so on relative to FP16):\n\n1. **Analytical Distributions and 46 Model-Zoo Networks + 3 LLMs:** On a standard normal distribution, INT4 quantization achieves a signal-to-noise ratio (SNR) of**19.1 dB** versus only**5.6 dB** for 75% magnitude pruning. Even when modeling heavy-tailed outlier distributions with a truncated Student’s- distribution () and measuring every weight tensor across**46 PyTorch model-zoo networks plus BLOOM-3B, LLaMA-3B, and OPT-2.7B** , uniform quantization achieves higher SNR than unstructured pruning for almost all tensors down to 4 bits; unstructured pruning only overtakes uniform quantization at extreme compression (**2 to 3 bits per value** ) on tensors with extreme kurtosis.\n2. **Algorithm-Independent Per-Layer Bounds:** To rule out the possibility that a better pruning heuristic could flip the ranking in post-training compression, Kuzmin et al. formulated per-layer output activation MSE minimization  as a Mixed-Integer Quadratic Program (MIQP). By comparing a convex Semi-Definite Programming (SDP) lower bound on quantization error against the**exact global optimum** of unstructured pruning (solved via Gurobi branch-and-bound across 10 layers of MobileNet-V2, EfficientNet-lite, ResNet-18, and ViT), they proved that quantization beats even globally optimal unstructured pruning at moderate compression ratios (4 to 8 bits) regardless of the optimizer used.\n3. **Full-Model QAT vs. Gradual Pruning with Fine-Tuning:** Across 9 models trained with equal fine-tuning epochs (LSQ quantization-aware training vs. gradual magnitude pruning), quantization won across the board. On**OPT-350M** (WikiText-103, baseline FP16 perplexity**14.8** ), QAT achieved perplexities of**14.8 (8b), 14.9 (6b), 15.3 (4b), 15.9 (3b), and 19.9 (2b)** , whereas gradual pruning with fine-tuning degraded to**18.0 (8b / 50% sparsity), 22.6 (6b), 35.4 (4b / 75% sparsity), 53.5 (3b), and 101.4 (2b)** .\nAs Kuzmin et al. note, because unstructured pruning has strictly more degrees of freedom than semi-structured 2:4 pruning,**unstructured pruning is an upper bound on the accuracy of 2:4 pruning** .\n\nNow move from idealized unstructured pruning to hardware **2:4 sparsity** and count the metadata bits honestly. In hardware, every kept value in an element-wise 2:4 tensor must store a **2-bit position index** specifying where it sat inside its 4-element group. For a group of 4 original elements, 2:4 stores 2 kept values plus  index bits—which is **1 metadata bit per original element**. If a dense symmetric integer format  with one FP16 scale per  elements costs  bits per element, then a 2:4 tensor whose kept values are quantized to  bits costs  value bits  index bit  scale bits per original element:\n\nHere is what that identity means for every format pairing:\n\n| Storage / Compute Tier | **2:4 Sparse Format** | **True Bits / Original Element***(incl. index & scale)* | **Equal-Bit or Equal-Tier Dense Rival** | **True Bits / Original Element** | \n|---|---|---|---|---|\n| **9 / 8-bit Budget ( Tier)** | **2:4 + FP16 / BF16** |  **()** | **Dense FP8** (E4M3)**Dense INT8** () |  | \n| **5-bit Budget ( Tier)** | **2:4 + FP8 / INT8** |  **()** | **Dense NVFP4** (, E4M3 scale)**Dense INT5** () |  | \n| **4-bit Budget** | **2:4 + INT6** |  **()** | **Dense INT4** ()**Dense MXFP4** (, E8M0 scale) |  | \n| **3-bit Budget** | **2:4 + INT4** |  **()** | **Dense INT3** () |  | \n| **Blackwell FP4 Floor ( Tier)** | **Pairwise 4:8 + NVFP4** |  **()** | *No dense 2.75-bit TC format exists* | — | \n\nDropping half the elements does not save half the bits: **2:4 + FP16 (9.0 bits) actually uses more memory than dense FP8 (8.0 bits)**, and **2:4 + FP8 (5.0 bits) uses more memory than dense NVFP4 (4.5 bits)**. At equal bits, the real choice is: *do you drop 2 of every 4 elements and quantize the surviving 2 elements with  bits each, or keep all 4 elements and quantize every one of them with  bits?*\n\nNow derive the error ratio  between those two choices.\nLet  be any of the six GEMM operands in a Transformer layer (), partitioned into groups  of 4 consecutive elements along the **hardware-forced reduction axis** of its GEMM:\n\n- **Linear weights  ():** grouped along input features;\n- **Linear activations  ():** grouped along hidden dimension;\n- **Queries  () and Keys  ():** grouped along the head dimension ;\n- **Values  ():** grouped along the token sequence axis;\n- **Attention probabilities  ():** grouped along the key sequence axis.\n\nIn each 4-element group , magnitude 2:4 pruning drops the two smallest-magnitude entries  and keeps the two largest entries . The relative Frobenius error energy of the pruning step alone is the **2:4 energy loss**:\n\nWhen the kept entries  are subsequently quantized to  bits with relative error energy  (normalized by ), notice that the pruning error lives solely on the dropped coordinates , while the quantization error lives solely on the disjoint kept coordinates . Because the two error vectors have disjoint support, they are **strictly orthogonal**, so their squared Frobenius norms add exactly. Comparing against a dense -bit quantizer with relative error energy  gives the exact tensor error ratio:\n\nThat inequality explains, without a single fitted parameter, why 2:4 loses above the quantization floor:\n\n1. **Quantization error shrinks exponentially with bit-width; 2:4 energy loss is a fixed floor.** Every time you add 1 bit to a uniform or floating-point quantizer, the step size  halves and the error energy  drops by roughly a factor of**does not depend on bit-width  at all** . No matter how many bits you spend on the two kept values , you can never recover a single bit of the energy  that 2:4 zeroed out.\nFor i.i.d. Gaussian data, order-statistic integration gives  ( relative RMSE):\n  - At **9 / 8.1 bits** (2:4 + FP16 vs. dense INT8): 2:4 error is **higher RMSE** ).\n  - At **5.1 bits** (2:4 + INT8 vs. dense INT5): 2:4 error is **higher RMSE** ).\n  - At **4.1 bits** (2:4 + INT6 vs. dense INT4): 2:4 error is **higher RMSE** ).\n  - At **3.1 bits** (2:4 + INT4 vs. dense INT3): 2:4 error is **higher RMSE** ).\nMoreover, unlike Kuzmin et al.’s global kurtosis predictor (which works for unstructured pruning of whole tensors, but only achieves a Spearman rank correlation of  on 2:4 because it ignores the hardware-forced grouping axis),  ranks the 3,400 measured  points for  with a**Spearman correlation of  and an AUC of** .\n2. At \n3. **At equal Tensor Core compute tiers ( and ), 2:4 is even further behind:** In a GEMM , the  tier pairs 2:4-FP16 on  (with dense FP16 ) against dense 8-bit on*both* and , and the  tier pairs 2:4-FP8 on  (with dense FP8 ) against dense 4-bit on*both* :\nBecause throwing away 2 of every 4 elements in  ( energy loss) is two orders of magnitude more destructive than rounding  and  to 8-bit FP8 ( energy loss), 2:4 gets blown out in the  tier and still loses in the  tier against NVFP4.\n4. **Why Hadamard Rotations (QuaRot, SpinQuant, ZYFA) Help Quantization and Hurt 2:4:** In modern low-precision inference (including ZYFA’s FP8 attention on MI355X, where applying a Hadamard rotation to  and  halves input quantization error at just  runtime cost, as well as QuaRot and SpinQuant), an orthogonal Hadamard transform  is applied before quantization: .\nWhat does  do to our inequality? By mixing channels,  wipes out channel outliers and makes every 4-element group look i.i.d. Gaussian. That**shrinks the quantization error** while**increasing** by spreading energy evenly across all four elements of each group—**roughly doubling the error ratio  against 2:4 for , , and** .\n\n### 4.2 Why does perplexity hide what pruning breaks?[#](#42-why-does-perplexity-hide-what-pruning-breaks)\n\nCompare the measured operator error ratio  ( means quantization wins) across all 32–36 layers of **Llama-3.1-8B-Instruct** and **Qwen3-8B** (221,408 configurations in total):\n\n| Operand Family & Forced Axis | **Storage: 8.1b**(vs. FP8-row) | **Storage: 5.1b**(vs. NVFP4-g16) | **Storage: 4.1b**(vs. INT4 / MXFP4) | **Storage: 3.1b**(vs. INT3-g128) | **Compute:  Tier**(2:4-FP16 vs. **FP8** ) | **Compute:  Tier**(2:4-FP8 vs. **NVFP4** ) | \n|---|---|---|---|---|---|---|\n|  **(Linear Weights, in-features)*** Llama-3.1-8B / Qwen3-8B* | / | / | / | / |  **/**  |  **/**  | \n|  **(Linear Inputs, hidden dim)*** Llama-3.1-8B / Qwen3-8B* | — | — | — | — |  **/**  |  **/**  | \n|  **(Queries, head dim )*** Llama-3.1-8B / Qwen3-8B* | — | — | — | — |  **/**  |  **/**  | \n|  **(Keys, head dim )*** Llama-3.1-8B / Qwen3-8B* | / | / | / | / |  **/**  |  **/**  | \n|  **(Values, token axis)*** Llama-3.1-8B / Qwen3-8B* | / | / | / | / |  **/**  |  **/**  | \n|  **(Attn Probs, key axis)*** Llama-3.1-8B / Qwen3-8B* | — | — | — | — |  **/**  |  **/**  | \n\nEvery single cell in that table is : **against FP8 at the  tier and NVFP4 at the  tier, 2:4 loses on every operand of both models, with  to  higher error than FP8 and  to  higher error than NVFP4.**\n\nIf 2:4 loses so decisively on operator error, why did so many pruning papers claim “negligible accuracy loss” at 50% sparsity? Because they evaluated compressed models almost exclusively on **WikiText-2 perplexity**—and four independent studies have now demonstrated that perplexity systematically hides the capabilities that pruning destroys:\n\n1. **LLM-KICK:** Evaluating Magnitude, Wanda, SparseGPT (both unstructured and 2:4 / 4:8 N:M structured sparsity), and GPTQ / AWQ / LLM.int8() across Vicuna-7B/13B/33B and LLaMA-1/2 on knowledge-intensive tasks, Jaiswal et al. showed that while WikiText-2 perplexity stays nearly flat up to 45%–50% sparsity,**FreebaseQA factoid question answering drops below the 5% tolerance threshold at trivial 25%–30% unstructured sparsity** (falling from**65.44%** dense on Vicuna-7B to**44.66%** under Wanda and**42.86%** under SparseGPT at 50% sparsity, while MMLU drops from**47.1%** to**38.6%** and**30.8%** ). Across*every* task in LLM-KICK—FreebaseQA, MMLU, TriviaQA, CNN/DailyMail summarization, and MT-Bench instruction following—**not a single N:M (2:4 or 4:8) sparse model achieved matching performance** , suffering  relative degradation on Factoid-QA. By contrast, 8-bit quantization (LLM.int8(), AWQ, GPTQ) and 4-bit GPTQ (on 13B) preserved MMLU (**47.10%–47.82%** ), retrieval-augmented QA (**75.55%–75.91%** ), and MT-Bench GPT-4 judge scores (**9.72–9.81** ). Even more tellingly, at an identical 7B active parameter count,**a smaller dense Vicuna-7B (46.7% MMLU) outperformed a 46.16%-pruned Vicuna-13B** (31.7% Magnitude, 45.3% Wanda, 46.3% SparseGPT).\n2. **Fact Recall Deteriorates Long Before In-Context Learning:** Why do pruned models look decent on next-token perplexity or short-context reading comprehension while failing on knowledge-intensive tasks? Jin et al. disentangled**in-weight parametric fact recall** (closed-book TriviaQA and WebQuestions) from**in-context learning (ICL)** (open-book TriviaQA, NaturalQuestions, counterfactual overriding**DisentQA** , and in-context learning of parameterized linear, 2-layer NN, and decision-tree classifiers) across OPT-13B/30B, LLaMA-13B/33B, and Pythia-12B under SparseGPT and Wanda. Accepting a 5% relative accuracy drop,**closed-book fact recall breaks at just 20%–30% sparsity** (30% on TriviaQA, 20% on Pythia-12B), whereas in-context copying and function learning survive up to**60%–70% sparsity** (50%–60% on open-book QA,**70% on counterfactual DisentQA** , and**60%–70% on in-context function learning** ). Furthermore, ablating module-level sparsity on LLaMA-13B showed that while attention and FFW layers are equally important for in-context learning,**pruning 60% of FFW layers degrades closed-book fact recall by 14 percentage points more than pruning 60% of attention layers** .\n3. **The “Junk DNA” Hypothesis:** Conventional pruning rests on the assumption that small-magnitude weights are redundant “junk”. By constructing six continuous task-difficulty spectra (varying target data volume on GLUE, option count from 2 to 4 on MMLU, context length on TriviaQA, -shot demonstrations  on MMLU, the normalized human-LLM gap across PIQA [17.5%], HellaSwag [37.6%], and OpenBookQA [60.4%], and FreebaseQA vs. MMLU), Yin et al. proved that**pruning small weights monotonically impairs downstream accuracy in direct proportion to task difficulty** across Magnitude, SparseGPT, and Wanda in both unstructured and N:M modes (Spearman rank correlation**GPTQ quantization at matched compression ratios does not exhibit this monotonic difficulty collapse** (Spearman ). Worse, Linear Mode Connectivity (LMC) interpolation and continual fine-tuning experiments (comparing Dense Transfer, Dense Transfer with 70%–90% of small weights frozen, Sparse Transfer, and Sparse-to-Dense Transfer where pruned weights are allowed to regrow) showed that**removing small pre-trained weights destroys the dense loss basin on hard tasks, causing irreparable accuracy loss even when pruned weights are allowed to regrow during fine-tuning** .\n4. **Spiky  vs. Non-Spiky  and Multi-Step Reasoning:** Inside attention itself, notice that  is the one operand in Qin’s table that comes closest to parity in the  tier ( against NVFP4, and  against weaker rivals like MXFP4 or row-wise INT8). After exponentiation, a spiky attention row concentrates most of its probability mass on one or two keys per 4-key group, so keeping the top 2 of every 4 logits in  acts like a local temperature sharpener—zeroing out background noise and slightly improving RULER 16K single-needle retrieval over row-wise INT8 and MXFP4.**Why is 2:4 on  (or ) still a capability trap?**  - First, **when attention is relatively flat or diffuse** —in early layers, induction/aggregation heads, or across multi-clause evidence—dropping 2 of every 4 logits in a contiguous 4-key window deletes up to**50% of the probability mass** whenever 3 or 4 relevant keys land in the same 4-token group.\n  - Second, even on the 15 boundary cells most favorable to 2:4 (), **2:4 never significantly beats quantization on GSM8K multi-step chain-of-thought reasoning** , and is significantly worse in 7 of those 15 cells (including 3 of the 5 cells where 2:4 won at the single-operator level). Single-needle retrieval only needs the argmax peak to survive, whereas a 256-token mathematical reasoning chain compounds every deleted secondary attention weight step after step across all 32 layers—an instance of the exact task-difficulty fragility identified by the Junk DNA Hypothesis.\n  - Third, if you also prune  or  along the head dimension  to escape the  single-sided Amdahl ceiling (§3.4), **query and key vectors are dense rotary/distributed embeddings along the head dimension—they are not spiky** . Even in the single most favorable cell for  (4× tier against weak MXFP4, where the*median* layer had ), Layers 0 and 1 of Llama-3.1-8B suffered **to  higher error** under 2:4 pruning. Because errors in Layer 0 and Layer 1 corrupt the residual stream for the rest of the network, whole-model WikiText-2 perplexity jumped from  to**more than 10 percentage points** .\n5. First, \n\n### 4.3 What did the field try after 2:4?[#](#43-what-did-the-field-try-after-24)\n\nBy 2024–2026, the compression community itself recognized that NVIDIA’s fixed 50% 2:4 pattern sits in a dead zone: too aggressive to preserve LLM reasoning without massive retraining, yet too rigid and low-speedup () to justify the accuracy loss. Researchers went in two opposite directions to work around 2:4’s hardware rigidity, and both say something about the underlying primitive:\n\n1. **Going Sparser via  Sparsity (*Beyond 2:4*):** Building on VENOM, Zhao et al. noted that 2:4 delivers only  practical speedup and cannot accelerate any other 50% pattern (like 4:8 or 8:16) or any ratio above 50%. In  sparsity, each weight matrix is partitioned into  blocks (); within each block,  entire columns are pruned away, and 2:4 sparsity is applied to the remaining 4 columns so the compressed block can still run on NVIDIA’s 2:4 Sparse Tensor Cores.\nWhile  (75% sparsity) achieved nearly lossless accuracy on vision backbones (DeiT-base on ImageNet-1K after 600 epochs of dynamic-mask SR-STE retraining,  vs.  dense), here is what happened when Zhao et al. applied  sparsity (60% sparsity) to**Llama2-7B** using -specific channel permutations (Hungarian algorithm) and a 3-stage LoRA fine-tuning recipe over 50,000 iterations on WikiText-2 and Alpaca:\n  - On RTX 3090,  lifted end-to-end speedup from  **(2:4)** to  - However, **WikiText-2 perplexity degraded from  (dense) to** (compared to  for training-free 2:4 SparseGPT/Wanda/RIA), and**8-task 5-shot average accuracy dropped by 8.95 points, from  (dense) to** (vs.  for Wanda 2:4). Even after 50,000 LoRA fine-tuning steps, 2:4 and  lost nearly 9 points of downstream capability.\n2. On RTX 3090,  lifted end-to-end speedup from \n3. **Going Denser via  Sparsity (*SlideSparse*):** At the opposite end, Shao et al. tested what happens to modern reasoning models under 2:4 pruning. Fine-tuning**Qwen3-1.7B** under identical settings in Dense, 6:8 (25% pruning), and 2:4 (50% pruning) across GSM8K, MATH-500, and HumanEval exposed a catastrophic cliff:\n  - **Dense Qwen3-1.7B:** GSM8K,  MATH-500,  HumanEval ( **average** ).\n  - **2:4 Sparse (50% pruned):** **GSM8K,  MATH-500,  HumanEval ( average—a 38.7-point collapse)** .\n  - **6:8 Sparse (25% pruned):** **GSM8K,  MATH-500,  HumanEval ( average)** .\nBecause NVIDIA Sparse Tensor Cores and`cuSPARSELt` offer**zero hardware support** for 6:8 (or any  pattern with ), inference engines must fall back to dense execution. To unlock Sparse Tensor Cores for  weights without hardware changes,*SlideSparse* introduced**Sliding Window Decomposition** : decomposing each -element weight block (containing at most  non-zeros) into  overlapping stride-2 windows of width 4, each holding at most 2 non-zeros (so for 6:8 with , 8 elements become  windows of 4 =**12 elements** ), while**Activation Lifting** duplicates the corresponding input activations inside a fused Triton per-token INT8/FP8 quantization kernel.\nThe arbitrage has a price:\n  - Expanding  elements into  elements inflates the GEMM reduction dimension  by the **expansion factor** :\n  - Dividing the  Sparse Tensor Core throughput by  **caps the theoretical maximum speedup at the inverse density** :", "url": "https://wpnews.pro/news/the-softmax-wall-and-the-2-4-sparse-attention-trap", "canonical_source": "https://vincentzed.com/writing/softmax-wall", "published_at": "2026-11-08 00:00:00+00:00", "updated_at": "2026-11-09 01:30:39.031339+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-chips", "machine-learning", "ai-research"], "entities": ["AMD", "NVIDIA", "Zephyr", "ZYFA", "Instinct MI355X", "Instinct MI455X", "Blackwell Ultra", "Vera Rubin"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-softmax-wall-and-the-2-4-sparse-attention-trap", "markdown": "https://wpnews.pro/news/the-softmax-wall-and-the-2-4-sparse-attention-trap.md", "text": "https://wpnews.pro/news/the-softmax-wall-and-the-2-4-sparse-attention-trap.txt", "jsonld": "https://wpnews.pro/news/the-softmax-wall-and-the-2-4-sparse-attention-trap.jsonld"}}