A researcher replaced the standard scaled dot-product attention in vision transformers with SSOG (Separable Sum of Gaussians), a content-blind geometric attention mechanism that uses a few Gaussian at…
PyTorch 2.13, released July 8, brings FlexAttention to Apple Silicon with up to 12x speedup on sparse attention patterns, such as a 32,768-token sequence with a 256-token sliding window (35ms vs 431ms…
A developer benchmarked PyTorch 2.13's FlexAttention on an M1 Max Mac, finding up to 7.83x speedup over standard SDPA for sparse attention with 32,768 tokens and a 256-token local window. FlexAttentio…
A developer wrote a fused decode-attention kernel that ran 2.2× faster than the baseline in microbenchmarks, but when integrated into a HuggingFace `generate` call for an RL training loop, the decode …