Decay-Gated O(N) Causal Linear Attention with Fused Triton Kernel

A developer has open-sourced a Decay-Gated O(N) Causal Linear Attention architecture with fused Triton/CUDA kernels, aiming to bypass quadratic multi-head attention bottlenecks. The project includes a repository and research paper, and the developer seeks integration into Hugging Face Transformers and benchmark comparisons.

Hey Hugging Face Community, I’ve engineered and open-sourced a Decay-Gated O N Causal Linear Attention architecture featuring fused Triton/CUDA state accumulation kernels designed to bypass quadratic MHA bottlenecks. Key Specs & Benchmark Metrics: Repository & Research Paper: Would love to gather thoughts on integration into Hugging Face Transformers model classes or benchmark comparisons with existing linear models