SSOG: Near linear Visual-Attention that doesn't score but steers
A researcher replaced the standard scaled dot-product attention in vision transformers with SSOG (Separable Sum of Gaussians), a content-blind geometric attention mechanism that uses a few Gaussian atoms per head and a c…