Depth-Staggered Fibonacci Spacing for Sparse Attention: Static Schedules Beat Learned Dilation and Extrapolate Where Dense Attention Fails
Researchers at arXiv show that static per-layer Fibonacci spacing in sparse attention outperforms learned dilation and fixed schedules across 21 language models, achieving better perplexity and extrapolating to four time…