Video generation has become a major frontier in generative modeling, driven by large-scale data and increasingly scalable architectures. Among modern architectures, Diffusion Transformers (DiTs) have emerged as a dominant paradigm [1,2,3,4] by representing videos as spatio-temporal token sequences, enabling long-range interactions across frames and spatial regions, as well as flexible multimodal conditioning with text or audio. However, full self-attention scales quadratically with token count, making it increasingly expensive as spatio-temporal resolution and model size grow. Video sparse attention (VSA) [5,6] mitigates this cost by approximating full attention with a subset of informative token interactions, but its practical efficiency in both training and inference depends heavily on hardware-aware Triton kernel implementations.
MoE with little models