Triton-Based Optimization of Video Sparse Attention on ROCm

Researchers developed Triton-based optimizations for video sparse attention on AMD's ROCm platform, aiming to reduce the quadratic computational cost of self-attention in Diffusion Transformers for video generation. The work addresses efficiency bottlenecks in training and inference by implementing hardware-aware kernels that approximate full attention with selective token interactions.

Video generation has become a major frontier in generative modeling, driven by large-scale data and increasingly scalable architectures. Among modern architectures, Diffusion Transformers DiTs have emerged as a dominant paradigm 1,2,3,4 by representing videos as spatio-temporal token sequences, enabling long-range interactions across frames and spatial regions, as well as flexible multimodal conditioning with text or audio. However, full self-attention scales quadratically with token count, making it increasingly expensive as spatio-temporal resolution and model size grow. Video sparse attention VSA 5,6 mitigates this cost by approximating full attention with a subset of informative token interactions, but its practical efficiency in both training and inference depends heavily on hardware-aware Triton kernel implementations.