{"slug": "nunchux-ai-introduces-vc-attention-a-training-free-low-bit-attention-kernel-that", "title": "Nunchux AI Introduces VC-Attention: A Training-Free Low-Bit Attention Kernel That Speeds Up Video Diffusion Transformers", "summary": "Nunchux AI released VC-Attention, a training-free low-bit attention kernel for video Diffusion Transformers that delivers up to 3.58x attention speedup and 1.70x end-to-end speedup on the RTX 5090 at 4-bit precision. The kernel combines V-Smooth, which clusters and demeans value tokens to cut quantization error, with ExpCast-FP8, which writes softmax bytes directly from the log-domain score; on B200 it runs 6.02x faster than SageAttention2, which ships no Blackwell kernel. Benchmarks cover four open-weight video DiTs — Wan2.2-T2V-A14B, LongCat-Video, HunyuanVideo-1.5, and MiniMax-H3 — with fidelity scored against BF16 FlashAttention-4 over 100 prompts.", "body_md": "Nunchux AI has released [VC-Attention](https://arxiv.org/abs/2609.15810), a training-free low-bit attention kernel built for video Diffusion Transformers (DiTs). It targets 2 problems at once: value quantization error and a slow softmax stage.\n\n## **Why Attention is the Video Bottleneck**\n\nVideo DiTs flatten a clip into 1 sequence of spatiotemporal tokens and run full self-attention at every layer. A 5-second 720p Wan2.2-14B clip spans about 70K tokens. On the RTX 5090, attention takes more than 64% of generation time. The research team states that attention is about two thirds of every MiniMax-H3 denoising step on a single B200.\n\nLow-bit Tensor Cores speed up the 2 matrix products, QK and PV. 2 obstacles remain. First, prior methods like [SageAttention2](https://arxiv.org/abs/2411.10958) smooth queries and keys. After QK smoothing and rotation, the value term accounts for 82% of output error on Wan2.2. Second, the softmax between the products still runs in FP32. On B200 and H200, that exponential and its FP8 cast become the longest pipeline stage.\n\n## **V-Smooth: Fixing Value Outliers**\n\nValue outliers sit in a few tokens, and their channels shift across heads, layers, and steps. A Hadamard rotation preserves token norms, so it does not remove them. Rotating V changes value error by just 0.2%.\n\n**V-Smooth takes a different route:**\n\n- **Group:** An online k-means clusters value tokens per batch and head. Keys and values are permuted together, so non-causal attention output is unchanged.\n- **Demean:** Each 128-token hardware block subtracts its mean. Only the residual is quantized, using per-channel E4M3 at 8 bits or NVFP4 at 4 bits.\n- **Restore:** The mean is added back using the row sum online softmax already keeps. No second pass or extra buffer is needed.\n\nAveraged over 100 Wan2.2 heads, the block mean removes 8% of block energy in sequence order. It removes 12% under DeltaQuant’s static cube and 36% after sorting. Each mean costs 0.125 bit per value element.\n\nGrouping runs only on the first 25% of denoising steps. The permutation is reused across 4 adjacent steps. Averaged over the full schedule, grouping costs 3 to 4% of attention time.\n\n## **ExpCast-FP8: Removing the Softmax Bottleneck**\n\nAn E4M3 byte is already close to a logarithm of the value it stores. Read as an integer, it equals roughly 8 log2(v) + 56. So ExpCast-FP8 writes the byte directly from the log-domain score with 1 fused multiply-add. The constant β = -0.35 centers the leftover error, and no constant is fitted per model.\n\nThe direct path writes the same byte as the FP32 exponent-then-cast path on 79.6% of each doubling. Elsewhere it lands 1 code away. The paper proves a per-row total variation bound under 3.64%, plus any underflow tail. Across 204.8K Wan2.2 attention rows, the measured average is 1.6%. ExpCast-FP8 applies only to the 8-bit kernel, since NVFP4 has no single affine log-to-code map.\n\nHand-written CuTe/CUDA fusion of the preprocessing chain cuts 1 V-Smooth call from 42.2 ms to 4.8 ms on B200.\n\n## **Explainer: How VC-Attention Works**\n\n## **Benchmarks**\n\nTests cover 4 open-weight video DiTs: Wan2.2-T2V-A14B, LongCat-Video, HunyuanVideo-1.5, and MiniMax-H3. Fidelity is scored against BF16 [FlashAttention-4](https://proceedings.mlsys.org/paper_files/paper/2026/hash/ae8b0b5838ba510daff1198474e7b984-Abstract-Conference.html) outputs over 100 prompts.\n\n| GPU (Wan2.2) | Precision | Attention speedup | End-to-end speedup | \n|---|---|---|---|\n| B200 | 8-bit | 1.59× | 1.19× | \n| H200 | 8-bit | 1.46× | 1.13× | \n| RTX PRO 6000 | 4-bit | 2.27× | 1.36× | \n| RTX 5090 | 4-bit | 3.58× | 1.70× | \n\nOn B200, VC-Attention is 6.02× faster than SageAttention2, which ships no Blackwell kernel. On H200, the gap is 1.16×. On workstation cards, 4-bit V-Smooth matches [SageAttention3](https://github.com/thu-ml/sageattention) on the RTX PRO 6000. It stays within 5% on the RTX 5090, so fidelity separates them.\n\n**Fidelity results:**\n\n- At 8 bits, V-Smooth adds 2.3 dB PSNR over SageAttention2 on Wan2.2 and 2.8 dB on HunyuanVideo-1.5.\n- Adding ExpCast-FP8 gives back 0.7 to 2.1 dB but still beats SageAttention2 on all 4 models.\n- At 4 bits, V-Smooth beats SageAttention3 by 2.9 dB on Wan2.2 and 3.6 dB on LongCat-Video.\n- Run training-free, [Attn-QAT](https://arxiv.org/abs/2603.00040) falls 3.4 to 6.7 dB below SageAttention2.\n\nOn MiniMax-H3 at 1344×768, attention runs 1.60× faster than BF16 FlashAttention-4 on B200. PSNR is 20.2 dB versus 19.9 dB for SageAttention2. On B300, the paper reports 1.47× versus 1.31× for a naive FP8 kernel. The blog chart lists 1.51× for B300.\n\nNunchux Attention, the company's proprietary extension, reaches 1.91× on B200 and 1.83× on B300 for MiniMax-H3 attention.\n\nThe method changes only per-interaction cost. So it can compose with sparse attention like [Sparse VideoGen](https://arxiv.org/abs/2502.01776) and [Radial Attention](https://arxiv.org/abs/2506.19852), and distillation and multi-GPU execution. Nunchux says free MiniMax-H3 access is coming through its [Modelverse waitlist](https://www.nunchux.ai/waitlist).\n\n## **Key Takeaways**\n\n- VC-Attention is training-free low-bit attention for video DiTs from Nunchux AI.\n- V-Smooth clusters value tokens, then quantizes only residuals after block-mean subtraction.\n- ExpCast-FP8 replaces the FP32 exponential and cast with 1 multiply-add.\n- Wan2.2 attention runs 1.59× faster on B200 and 3.58× on RTX 5090.\n- No public kernel release yet; Nunchux runs a proprietary extension in its stack.\n\nCheck out the **[Paper](https://arxiv.org/pdf/2609.15810)** and [**Technical details**](https://github.com/jmiao24/Paper2Agent). All credit goes to the researcher of this project. Also, feel free to follow us on **[Twitter](https://x.com/intent/follow?screen_name=marktechpost)** and don’t forget to join our **[150k+ML SubReddit](https://www.reddit.com/r/machinelearningnews/)** and Subscribe to **[our Newsletter](https://magic.beehiiv.com/v1/f5e63dd4-5653-4f09-83e2-321a8b1ba526?email={{email}})**. Wait! are you on telegram? [now you can join us on telegram as well.](https://t.me/machinelearningresearchnews)\n\nNeed to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? [Connect with us](https://forms.gle/wbash1wF6efRj8G58)\n\nAsif Razzaq is the CEO of Marktechpost AI Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.", "url": "https://wpnews.pro/news/nunchux-ai-introduces-vc-attention-a-training-free-low-bit-attention-kernel-that", "canonical_source": "https://www.marktechpost.com/2026/09/16/nunchux-ai-introduces-vc-attention-a-training-free-low-bit-attention-kernel-that-speeds-up-video-diffusion-transformers/", "published_at": "2026-09-17 00:45:06+00:00", "updated_at": "2026-09-17 00:52:54.098950+00:00", "lang": "en", "topics": ["artificial-intelligence", "generative-ai", "ai-research", "machine-learning", "ai-infrastructure"], "entities": ["Nunchux AI", "VC-Attention", "V-Smooth", "ExpCast-FP8", "SageAttention2", "Wan2.2-T2V-A14B", "MiniMax-H3", "FlashAttention-4"], "alternates": {"html": "https://wpnews.pro/news/nunchux-ai-introduces-vc-attention-a-training-free-low-bit-attention-kernel-that", "markdown": "https://wpnews.pro/news/nunchux-ai-introduces-vc-attention-a-training-free-low-bit-attention-kernel-that.md", "text": "https://wpnews.pro/news/nunchux-ai-introduces-vc-attention-a-training-free-low-bit-attention-kernel-that.txt", "jsonld": "https://wpnews.pro/news/nunchux-ai-introduces-vc-attention-a-training-free-low-bit-attention-kernel-that.jsonld"}}