SSOG: Near linear Visual-Attention that doesn't score but steers

A researcher replaced the standard scaled dot-product attention in vision transformers with SSOG (Separable Sum of Gaussians), a content-blind geometric attention mechanism that uses a few Gaussian atoms per head and a content-conditioned steering nudge, eliminating the N×N attention matrix for near-linear scaling. On ImageNet, the fixed field alone came within one point of SDPA, steering closed the gap, and the full model beat the baseline; a 12M version reached 72% accuracy, 20% smaller and 30% cheaper, and on small data the geometric prior gave a +17 point boost.

A Few Gaussians Is All You Need: ../../posts/ssog/ SSOG-Attention That Steers Instead of Scores SSOG-Attention That Steers Instead of Scores I swapped the transformer's matchmaking service for a handful of Gaussians, and it beat content scoring without comparing a single pair of tokens. Crack open any vision transformer and you'll find the same machinery humming at its core: scaled dot-product attention SDPA . Every token asks every other token "how much do I care about you?", and the answer is a big $N \times N$ matrix of similarity scores, computed from the content of the tokens themselves. We rarely second-guess this. I know, because I tried once before: a while ago I swapped the dot product for an RBF kernel ../../posts/scaled-rbf-attention/ and learned a lot about what similarity even means in these systems. This time I wanted to question something more fundamental: Why does the network need to compute "where to look" from scratch, for every image, from content? Think about how you read an image. When you process a patch on a bird's wing, you don't run a similarity search against all other patches. You already know where to look: a bit left, a bit up is that the head? , further out where does the wing end? . Where to look is mostly a function of geometry , not content. Content only fine-tunes it. So I built attention that way. Each head owns a few Gaussians, a small handful of numbers in total, forming a fixed field over relative position. A learned habit of where to look. Then the trick that makes it actually work: a tiny content-conditioned nudge that lets each token shift its field. No query-key dot products anywhere. Content never scores, it only steers. The results surprised me. The fixed field alone, completely content-blind, comes within one point of SDPA on ImageNet. Steering closes that gap entirely, and the full model beats the baseline. On small data the geometric prior is worth a ridiculous +17 points . It scales too: a 12M version reaches 72% on ImageNet, ahead of its SDPA twin while being 20% smaller and 30% cheaper to run. The kicker: since the field factorizes into two 1D filter passes, the $N \times N$ attention matrix never exists, so this is attention that scales near-linearly instead of quadratically. Best of all, "what did attention learn?" stops being a heatmap-and-a-shrug question. Every head is just a few blobs you can literally plot and read with a ruler. a score every pair from content. b a fixed Gaussian field, same for every image. c content only steers that field. ★ = query. Meet the Field This is the entire attention mechanism. Each head owns a few Gaussian atoms. An atom is five numbers: a center offset μy, μx , a width in each direction σy, σx , and a weight λ. How many atoms per head is a free dial even one gets you most of the way . ★ is the query. μ is where an atom looks, σ how wide it stares, λ how much it counts. Three atoms = fifteen numbers. That's the head: a handful of numbers instead of an $N \times N$ matrix of content-dependent scores. The weight from "here" to "there" is just this mixture evaluated at the displacement between them: $$A p, q = \mathrm{softmax} q \Big \tfrac{1}{\tau}\; s p, q \Big $$ with each atom contributing its log-weighted Gaussian to the score: $$s p, q = \mathrm{logsumexp} r \big \log \lambda r + \log \mathcal{N} p - q;\ \mu r, \sigma r \big $$ Same field for every image, every time. A cat photo and a car photo get the exact same attention pattern. I called the family SSOG Separable Sum of Gaussians , because that's what it is. But a purely fixed field only gets you so far. Images aren't all the same, and sometimes the bird is just not in the center. So the field needs to move. The fix is called mu delta , with its siblings sigma delta and lambda gate . Each token gets one tiny linear layer zero-initialized that predicts small residuals on the field parameters: $$\begin{aligned} \mu &\leftarrow \mu 0 + s \mu \cdot 4 \cdot \tanh W \mu x \\ \sigma &\leftarrow \sigma 0 \cdot e^{s \sigma \tanh W \sigma x }\\ \lambda &\leftarrow \mathrm{softmax}\big \log \lambda 0 + s \lambda \tanh W \lambda x \big \end{aligned}$$ In words: content may shift where an atom looks, widen or tighten how hard it stares, and re-weight which atoms matter. All bounded, all starting at zero. I call it lookat . Tanh caps travel at $\pm$4 grid cells, and because the gates start at $\approx$ 0 the model begins as a frozen geometric animal and learns whether to open the content taps at all . Spoiler: every layer opens them. Plot coming. Build a head yourself: drag atoms, widen them, re-weight them, or watch content steer onto the bird. The presets are geometries the trained model actually converged to: If this sounds crazy: it kind of is, and it kind of isn't. Synthesizer https://arxiv.org/abs/2005.00743 showed in 2020 that even random learned attention matrices work surprisingly well. I'm just forcing the matrix to be a smooth, translation-invariant geometric object — a spatial prior, like convolutions, but softer and longer-ranged. The Separable Trick a.k.a. How to Never Build the $N \times N$ Matrix One more piece, and the reason any of this is fast. A 2D Gaussian factors: $\mathcal{N} \Delta y, \Delta x = \mathcal{N} \Delta y \cdot \mathcal{N} \Delta x $. Applying the field is two 1D filter passes: Row pass × column pass = one atom's 2D field. ★ = query. Dense attention builds an $N \times N$ map $O N^{2} d $ ; separable SSOG never does $O N \sqrt{N}\, d $ per atom . Filter rows, then columns, once per atom; mix with λ; done. Three einsums per layer: php y = jnp.einsum "biwprj,bjwpd- biwpdr", ay, v down the rows, per atom y = jnp.einsum "biwprk,bikpdr- biwpdr", ax, y across the columns y = jnp.einsum "pr,biwpdr- biwpd", lam, y mix atoms Factorized mode costs $O N \cdot \sqrt{N} \cdot d $ instead of $O N^2 \cdot d $ and loses nothing in accuracy. People chase "linear attention" with low-rank tricks and kernel approximations; this gets near-linear by geometry alone, because the field is a filter, not a matrix. There's also an "axial" variant that mixes in logit space; same cost, same results — I'll spare you the taxonomy. The Experiment I bolted this onto a small ViT dim 256, 6 layers, 4 heads, ~2.9M params and threw the usual soup at it: CIFAR-100 $32 \times 32$ and ImageNet-1k $224 \times 224$ , from scratch, same 90-epoch recipe, one RTX 5090. No TPU pods were harmed. The SDPA baseline is the same network with standard attention swapped back in. Part One, in Which My Baseline Face-Plants CIFAR-100 first, three seeds each I've been burned . The SSOG variants all landed around 70%, huddled like penguins: Then SDPA: 53%. I stared at that for a while. The diagnosis is obvious from the curves: SDPA overfits like crazy 92% train, 53% test and optimizes slowly from epoch one. At $32 \times 32$ with 500 images per class, content scoring doesn't have enough data to learn geometry from scratch. The Gaussians start with it. A fixed "look at your neighborhood" habit is worth ~17 points on small data. Fairness footnote, before the pitchforks: a longer 300-epoch recipe would let SDPA recover a lot of this. The claim is narrow on purpose: at matched recipe, the geometric prior is a superpower on small data. Part Two: ImageNet, Where Content Fights Back ImageNet-1k is the opposite regime: 1.28M images, $224 \times 224$, plenty of room for content scoring to shine. The fixed field should lose... and it does, by a point: ImageNet-1k, 90 epochs, matched recipe | Variant | μδ | σδ | λ-gate | Params | Val acc | Δ vs SDPA | |---|---|---|---|---|---|---| | Fixed field axial | ✗ | ✗ | ✗ | 2.88M | 63.21% | $-1.13$ | | Fixed field factorized | ✗ | ✗ | ✗ | 2.88M | 63.39% | $-0.95$ | | SDPA baseline | ✗ | ✗ | ✗ | 3.66M | 64.34% | $\pm$0.00 | | +μδ axial | ✓ | ✗ | ✗ | 2.93M | 64.45% $\pm$ 0.20 | +0.11 | | +μδ factorized | ✓ | ✗ | ✗ | 2.92M | 64.48% $\pm$ 0.13 | +0.14 | +μδ +σδ +λ factorized | ✓ | ✓ | ✓ | 3.00M | 65.28% $\pm$ 0.02 | +0.94 | $\pm$ is std over 3 seeds; bare rows are single runs. SDPA is the largest model here — those QK projections aren't free. Three takeaways. The fixed field, content-blind, lands one point behind SDPA . That the gap is this small still surprises me. Steering alone μδ closes it completely: 64.5% $\pm$ 0.2, dead-even with SDPA — so steering was all that was missing . Full conditioning beats content scoring, with seed variance so small 65.27 / 65.27 / 65.31 that I re-ran the eval twice to believe it. And the factorized rows, which never build $N \times N$, match axial everywhere. Free lunch, served. The geometric arms lead from the first epochs, while the spatial prior pays rent before content scoring has learned anything. SDPA closes most of the gap mid-schedule. It never quite catches up. But Does It Scale? Everything above was 2.9M params. The real test: a bigger sibling dim 384, 12 layers, 6 heads against its SDPA twin, same recipe, same ImageNet. SSOG finished at 72.02% +6.7 over the d256 champion , SDPA at 71.84% . Honest reading: the accuracy gap shrunk to a sliver +0.18, down from +0.94 . With 1.28M images, content scoring catches up on points. The efficiency margins are structural: SSOG stays ahead while being 20% smaller and 30% cheaper per forward, and the interpretability is free. Same story as d256, louder: SSOG jumps to a double-digit lead early, SDPA closes through the middle, they finish within a fifth of a point. After epoch one's coin flip, the geometric arm is never behind. Smaller, cheaper, better: the twins at two scales | Model | Params | FLOPs / forward | Val acc 90 ep | |---|---|---|---| | d256 $\cdot$ SDPA | 3.66M | ~1.5 G | 64.34% | | d256 $\cdot$ SSOG +μδσλ | 3.00M | ~1.0 G | 65.28% | | d384 $\cdot$ SDPA | 14.94M | ~6.3 G | 71.84% | | d384 $\cdot$ SSOG +μδσλ | 11.96M | ~4.4 G | 72.02% | At d256, "better" meant raw accuracy. At d384, it means you keep the accuracy and the geometry. Same image, same mechanism, twelve layers deep. The Ablation Drawer Every mechanism paper has a drawer of "we tried this knob." Here's mine CIFAR-100, single seeds, vs the 70.4% reference : Three favorites: - One Gaussian per head is almost enough. atom rank=1 loses only 0.7 points. A single deformable blob — the minimal mechanism — already works. Four is the sweet spot here; eight is showing off. Bigger models would probably put more atoms to work. - The cold start matters. Warm-init the gates and you lose 1.1 points. Starting geometric and learning to listen to content beats doing both from scratch. There's probably a cute optimization-dynamics story here. - The mechanism knows its regime. Small offsets win at small scale: $\pm$1–2 cells beat $\pm$4 on CIFAR $32 \times 32$; nowhere to go , while ImageNet wanted $\pm$4. And μδ is a null result on CIFAR-100 70.40% vs 70.39% but +1.2 on ImageNet : on a $16 \times 16$ grid the fixed field already covers everything worth covering; at $224^2$ there's room to move. Content-conditioned geometry pays exactly then. Party Tricks Two freebies that made me grin: Zero-shot resolution transfer. The field lives on coordinates , not token indices, so you train at $224^2$ 196 tokens and just... evaluate bigger. Only the tiny position embedding needs a bilinear resize; the Gaussians re-evaluate on the new grid. On the d384 champion, $288^2$ scores 73.7% — higher than the 72.0% at train resolution — with zero fine-tuning. Still 71.6% at $384^2$; at $512^2$ the $\pm$4-cell offset bound finally runs out of reach. Note for v2. Left: latency vs tokens at d384/L12/h6 batch 64, bf16; linear axes . Right: zero-shot transfer of the d384 factorized +μδσλ checkpoint — peak 73.7% at 288², then the offset bound bites. Speed that scales. Never building $N \times N$ pays rent. Geometry replaces the similarity matrix, so SSOG skips QK entirely: SDPA pays $4d^2$ per layer for Q, K, V, and the output proj; SSOG pays $2d^2$, plus a few percent for steering. At d256 that is ~1.0 GFLOPs/forward vs SDPA's ~1.5; at d384, ~4.4 vs ~6.3. From 196 → 1024 tokens on the bigger twin, SDPA slows ~12× while axial fixed slows ~7× and factorized +μδ sits in between per-query maps aren't free . The point stands: no $N^2$ anywhere . Margins only grow with scale. Opening the Hood Favorite part. We'll open the d384 champion. "What did attention learn?" usually gets a heatmap and a shrug. Here every head is a few Gaussians, so I can just show you the whole geometry — twelve layers, six heads: λ-weighted mixture density over relative position per panel normalized . ★ = query self ; + = atom centers brighter = larger λ . Layer 0 is a convolution in disguise: tight blobs around the query. Middle layers stretch into long vertical and horizontal bars — edge and strip detectors that span the image in one direction. Late layers go broad and global. All of it from random Gaussians plus gradient descent; nobody told layer 6 to become a line detector. And the gates? Every content tap starts frozen at $\approx$ 0. Here's how far each layer opened them: Learned gate scales after training. All three started at ≈ 0 dashed . Every layer, every gate: opened. Not equally. $s \lambda $ climbs hardest — late layers love re-weighting atoms — while $s \mu\delta $ stays modest and fairly flat. The model has opinions about which content taps matter. This is the figure I'd frame on my wall. Same bird as the playground, but instead of another pretty heatmap I asked a bookkeeping question: at each layer, how much does each patch pull in from the others, and how much is pulled out of it? With the residual-mixed mean-head matrix $\hat{A}$: $$\begin{aligned} \mathrm{pull} {\mathrm{in},i} &= 1 - \hat{A} {ii}, \\ \mathrm{pull} {\mathrm{out},i} &= \sum j \hat{A} {ji} - \hat{A} {ii}. \end{aligned}$$ Net export is their difference. Red = source others read it more than it reads them ; blue = gatherer. Sum over layers and you get the information economy of the whole forward pass: Top: input. Middle: per-layer net export RdBu, 0-centered . Bottom: accumulated over all 12 layers — range about −2.4, 7.5 , ~38% of patches net exporters. Early layers are almost balanced; everyone still mostly talks to themselves. Then the split opens: bird and post go red, background goes blue. By the last layers the silhouette is unmistakable, and the accumulated map is blunt about it. Only ~38% of patches are net exporters, and they are exactly the ones that matter. Everyone else is shopping there. That's the point of steering, as an accounting identity: content doesn't have to score against content. It only has to decide who gets to be the source. The field parked its exports on the subject. The background learned to shop. Is This New? Honest answer — I've been on the wrong side of this question: the ingredients are all published. Content-free learned attention is Synthesizer https://arxiv.org/abs/2005.00743 2020 . Position-only interaction is AFT https://arxiv.org/abs/2105.14103 and LambdaNetworks https://arxiv.org/abs/2102.08602 . Gaussians near attention are everywhere: positional biases, RBF kernels over embeddings guilty ../../posts/scaled-rbf-attention/ ; HALO ../../posts/halo/ aims the same geometry at classification heads , fixed weights for point clouds. Deforming attention with predicted offsets is Deformable-DETR https://arxiv.org/abs/2010.04159 / DAT https://arxiv.org/abs/2201.00520 territory. What I haven't found is the combo: a full attention operator that is 1 a Gaussian mixture over relative position, 2 applied separably without ever forming $N \times N$, and 3 deformed per-query through bounded, cold-started residuals on μ, σ, and λ. DAT keeps content scores and samples discrete points; this deforms a continuous field and never scores. Seen it published? Inbox is open. Prior-art diligence is a team sport. Would I Bet on It? Cautiously, yes. Honest limits: still small models 3M and 12M , one recipe, vision only, and the μδ maps add a memory term I'd like to engineer away. The real proof is bigger models on bigger data — that job I leave to the GPU-upper class; I'm GPU-poor over here. Language is a different beast; content scoring is probably load-bearing there, though I'd love to be wrong. The core finding still feels robust, and honestly surprising: on images, content doesn't need to score. It only needs to steer. Fixed geometry plus tiny bounded nudges matches SDPA on ImageNet, demolishes it on small data, scales near-linearly, transfers across resolutions for free, and turns "what did the model learn?" into a question you answer with a ruler. The Code The clean, minimal implementation is on GitHub: ssog/attention.py holds the Gaussian field and its SDPA twin, ssog/vit.py wires up a small ViT that runs with either, and examples/train cifar100.py trains both so you can compare them on your own GPU. Swapping attention mechanisms is a one-liner: model = ViT num classes=100, attn="ssog" the Gaussian field model = ViT num classes=100, attn="dot" its SDPA twin If you poke at the mechanism, break it on your own data, or prove me wrong about language models, I'd love to hear what you find. And a star on the repo is always appreciated.