{"slug": "ssog-near-linear-visual-attention-that-doesn-t-score-but-steers", "title": "SSOG: Near linear Visual-Attention that doesn't score but steers", "summary": "A researcher replaced the standard scaled dot-product attention in vision transformers with SSOG (Separable Sum of Gaussians), a content-blind geometric attention mechanism that uses a few Gaussian atoms per head and a content-conditioned steering nudge, eliminating the N×N attention matrix for near-linear scaling. On ImageNet, the fixed field alone came within one point of SDPA, steering closed the gap, and the full model beat the baseline; a 12M version reached 72% accuracy, 20% smaller and 30% cheaper, and on small data the geometric prior gave a +17 point boost.", "body_md": "#\n[\nA Few Gaussians Is All You Need:](../../posts/ssog/)\n\nSSOG-Attention That Steers Instead of Scores\n\nSSOG-Attention That Steers Instead of Scores\n\n*I swapped the transformer's matchmaking service for a handful of Gaussians,\nand it beat content scoring without comparing a single pair of tokens.*\n\nCrack open any vision transformer and you'll find the same machinery humming at its core: scaled dot-product attention (SDPA). Every token asks every other token \"how much do I care about you?\", and the answer is a big $N \\times N$ matrix of similarity scores, computed from the content of the tokens themselves.\n\nWe rarely second-guess this. I know, because I tried once before: [a while ago\nI swapped the dot product for an RBF kernel](../../posts/scaled-rbf-attention/) and learned\na lot about what similarity even *means* in these systems. This time I wanted\nto question something more fundamental:\n\n**Why does the network need to compute \"where to look\" from scratch, for every\nimage, from content?**\n\nThink about how *you* read an image. When you process a patch on a bird's wing,\nyou don't run a similarity search against all other patches. You already *know*\nwhere to look: a bit left, a bit up (is that the head?), further out (where\ndoes the wing end?). Where to look is mostly a function of **geometry**, not\ncontent. Content only fine-tunes it.\n\nSo I built attention that way. Each head owns a few Gaussians, a small handful\nof numbers in total, forming a fixed field over relative position. A learned\nhabit of where to look. Then the trick that makes it actually work: a tiny\ncontent-conditioned nudge that lets each token *shift* its field. No query-key\ndot products anywhere. Content never scores, it only steers.\n\nThe results surprised me. The fixed field alone, completely content-blind,\ncomes within **one point** of SDPA on ImageNet. Steering closes that\ngap entirely, and the full model *beats* the baseline. On small data the\ngeometric prior is worth a ridiculous **+17 points**. It scales too: a 12M\nversion reaches **72%** on ImageNet, ahead of its SDPA twin while being 20%\nsmaller and 30% cheaper to run. The kicker: since the field factorizes into two\n1D filter passes, the $N \\times N$ attention matrix never exists, so this is attention\nthat scales *near-linearly* instead of quadratically. Best of all, \"what did attention\nlearn?\" stops being a heatmap-and-a-shrug question. Every head is just a few\nblobs you can literally plot and read with a ruler.\n\n(a) score every pair from content. (b) a fixed Gaussian field, same for every\nimage. (c) content only *steers* that field. ★ = query.\n\n## Meet the Field\n\nThis is the entire attention mechanism. Each head owns a few Gaussian atoms. An atom is five numbers: a center offset (μy, μx), a width in each direction (σy, σx), and a weight λ. How many atoms per head is a free dial (even one gets you most of the way).\n\n★ is the query. μ is where an atom looks, σ how wide it stares, λ how much it counts. Three atoms = fifteen numbers.\n\nThat's the head: a handful of numbers instead of an $N \\times N$ matrix of content-dependent scores. The weight from \"here\" to \"there\" is just this mixture evaluated at the displacement between them:\n\n$$A(p, q) = \\mathrm{softmax}_q \\Big( \\tfrac{1}{\\tau}\\; s(p, q) \\Big)$$\n\nwith each atom contributing its log-weighted Gaussian to the score:\n\n$$s(p, q) = \\mathrm{logsumexp}_r \\big( \\log \\lambda_r + \\log \\mathcal{N}(p - q;\\ \\mu_r, \\sigma_r) \\big)$$\n\nSame field for every image, every time. A cat photo and a car photo get the exact same attention pattern. I called the family SSOG (Separable Sum of Gaussians), because that's what it is.\n\nBut a purely fixed field only gets you so far. Images aren't all the same, and\nsometimes the bird is just *not* in the center. So the field needs to move.\nThe fix is called `mu_delta`\n\n, with its siblings `sigma_delta`\n\nand\n`lambda_gate`\n\n. Each token gets one tiny linear layer (zero-initialized!) that\npredicts small residuals on the field parameters:\n\n$$\\begin{aligned} \\mu &\\leftarrow \\mu_0 + s_\\mu \\cdot 4 \\cdot \\tanh(W_\\mu x)\\\\ \\sigma &\\leftarrow \\sigma_0 \\cdot e^{s_\\sigma \\tanh(W_\\sigma x)}\\\\ \\lambda &\\leftarrow \\mathrm{softmax}\\big(\\log \\lambda_0 + s_\\lambda \\tanh(W_\\lambda x)\\big) \\end{aligned}$$\n\nIn words: content may **shift** where an atom looks, **widen or tighten** how\nhard it stares, and **re-weight** which atoms matter. All bounded, all starting\nat zero. I call it *lookat*. Tanh caps travel at $\\pm$4 grid cells, and because\nthe gates start at $\\approx$ 0 the model **begins as a frozen geometric animal and\nlearns whether to open the content taps at all**.\n\n(Spoiler: every layer opens them. Plot coming.)\n\nBuild a head yourself: drag atoms, widen them, re-weight them, or watch content steer onto the bird. The presets are geometries the trained model actually converged to:\n\nIf this sounds crazy: it kind of is, and it kind of isn't.\n[Synthesizer](https://arxiv.org/abs/2005.00743) showed in 2020 that even\n*random* learned attention matrices work surprisingly well. I'm just forcing\nthe matrix to be a smooth, translation-invariant geometric object — a spatial\nprior, like convolutions, but softer and longer-ranged.\n\n## The Separable Trick (a.k.a. How to Never Build the $N \\times N$ Matrix)\n\nOne more piece, and the reason any of this is fast. A 2D Gaussian factors: $\\mathcal{N}(\\Delta y, \\Delta x) = \\mathcal{N}(\\Delta y) \\cdot \\mathcal{N}(\\Delta x)$. Applying the field is two 1D filter passes:\n\nRow pass × column pass = one atom's 2D field. ★ = query. Dense attention builds an $N \\times N$ map ($O(N^{2} d)$); separable SSOG never does ($O(N \\sqrt{N}\\, d)$ per atom).\n\nFilter rows, then columns, once per atom; mix with λ; done. Three einsums per layer:\n\n``` php\ny = jnp.einsum(\"biwprj,bjwpd->biwpdr\", ay, v)   # down the rows, per atom\ny = jnp.einsum(\"biwprk,bikpdr->biwpdr\", ax, y)   # across the columns\ny = jnp.einsum(\"pr,biwpdr->biwpd\", lam, y)       # mix atoms\n```\n\nFactorized mode costs $O(N \\cdot \\sqrt{N} \\cdot d)$ instead of $O(N^2 \\cdot d)$ and loses\n*nothing* in accuracy. People chase \"linear attention\" with low-rank tricks and\nkernel approximations; this gets near-linear by geometry alone, because the\nfield is a filter, not a matrix. (There's also an \"axial\" variant that mixes in\nlogit space; same cost, same results — I'll spare you the taxonomy.)\n\n## The Experiment\n\nI bolted this onto a small ViT (dim 256, 6 layers, 4 heads, ~2.9M params) and\nthrew the usual soup at it: CIFAR-100 ($32 \\times 32$) and ImageNet-1k\n($224 \\times 224$), from scratch, same 90-epoch recipe, one RTX 5090. No TPU pods\nwere harmed. The SDPA baseline is the *same* network with standard attention\nswapped back in.\n\n## Part One, in Which My Baseline Face-Plants\n\nCIFAR-100 first, three seeds each (I've been burned). The SSOG variants all landed around 70%, huddled like penguins:\n\nThen SDPA: **53%.** I stared at that for a while.\n\nThe diagnosis is obvious from the curves: SDPA *overfits like crazy* (92% train,\n53% test) and optimizes slowly from epoch one. At $32 \\times 32$ with 500 images per\nclass, content scoring doesn't have enough data to learn geometry from scratch.\nThe Gaussians *start* with it. A fixed \"look at your neighborhood\" habit is\nworth ~17 points on small data.\n\nFairness footnote, before the pitchforks: a longer 300-epoch recipe would let\nSDPA recover a lot of this. The claim is narrow on purpose: **at matched recipe,\nthe geometric prior is a superpower on small data.**\n\n## Part Two: ImageNet, Where Content Fights Back\n\nImageNet-1k is the opposite regime: 1.28M images, $224 \\times 224$, plenty of room\nfor content scoring to shine. The fixed field *should* lose... and it does, by\na point:\n\n##### ImageNet-1k, 90 epochs, matched recipe\n\n| Variant | μδ | σδ | λ-gate | Params | Val acc | Δ vs SDPA |\n|---|---|---|---|---|---|---|\n| Fixed field (axial) | ✗ | ✗ | ✗ | 2.88M | 63.21% | $-1.13$ |\n| Fixed field (factorized) | ✗ | ✗ | ✗ | 2.88M | 63.39% | $-0.95$ |\n| SDPA (baseline) | ✗ | ✗ | ✗ | 3.66M | 64.34% | $\\pm$0.00 |\n| +μδ (axial) | ✓ | ✗ | ✗ | 2.93M | 64.45% $\\pm$ 0.20 | +0.11 |\n| +μδ (factorized) | ✓ | ✗ | ✗ | 2.92M | 64.48% $\\pm$ 0.13 | +0.14 |\n+μδ +σδ +λ (factorized) |\n✓ | ✓ | ✓ | 3.00M | 65.28% $\\pm$ 0.02 |\n+0.94 |\n\n$\\pm$ is std over 3 seeds; bare rows are single runs. SDPA is the *largest*\nmodel here — those QK projections aren't free.\n\nThree takeaways. The fixed field, content-blind, lands **one point behind\nSDPA**. That the gap is this small still surprises me. Steering alone (μδ)\ncloses it completely: 64.5% $\\pm$ 0.2, dead-even with SDPA — so steering was *all\nthat was missing*. Full conditioning *beats* content scoring, with seed\nvariance so small (65.27 / 65.27 / 65.31) that I re-ran the eval twice to\nbelieve it. And the factorized rows, which never build $N \\times N$, match axial\neverywhere. Free lunch, served.\n\nThe geometric arms lead from the first epochs, while the spatial prior pays rent before content scoring has learned anything. SDPA closes most of the gap mid-schedule. It never quite catches up.\n\n### But Does It Scale?\n\nEverything above was 2.9M params. The real test: a bigger sibling (dim 384,\n12 layers, 6 heads) against its SDPA twin, same recipe, same ImageNet. SSOG\nfinished at **72.02%** (+6.7 over the d256 champion), SDPA at **71.84%**. Honest\nreading: the accuracy gap shrunk to a sliver (+0.18, down from +0.94). With\n1.28M images, content scoring catches up on points. The efficiency margins are\nstructural: SSOG stays ahead while being 20% smaller and 30% cheaper per\nforward, and the interpretability is free.\n\nSame story as d256, louder: SSOG jumps to a double-digit lead early, SDPA closes through the middle, they finish within a fifth of a point. After epoch one's coin flip, the geometric arm is never behind.\n\n##### Smaller, cheaper, better: the twins at two scales\n\n| Model | Params | FLOPs / forward | Val acc (90 ep) |\n|---|---|---|---|\n| d256 $\\cdot$ SDPA | 3.66M | ~1.5 G | 64.34% |\n| d256 $\\cdot$ SSOG +μδσλ | 3.00M |\n~1.0 G |\n65.28% |\n| d384 $\\cdot$ SDPA | 14.94M | ~6.3 G | 71.84% |\n| d384 $\\cdot$ SSOG +μδσλ | 11.96M |\n~4.4 G |\n72.02% |\n\nAt d256, \"better\" meant raw accuracy. At d384, it means you keep the accuracy\n*and* the geometry. Same image, same mechanism, twelve layers deep.\n\n## The Ablation Drawer\n\nEvery mechanism paper has a drawer of \"we tried this knob.\" Here's mine (CIFAR-100, single seeds, vs the 70.4% reference):\n\nThree favorites:\n\n-\n**One Gaussian per head is almost enough.** atom_rank=1 loses only 0.7 points. A single deformable blob — the minimal mechanism — already works. Four is the sweet spot here; eight is showing off. Bigger models would probably put more atoms to work. -\n**The cold start matters.** Warm-init the gates and you lose 1.1 points. Starting geometric and*learning*to listen to content beats doing both from scratch. There's probably a cute optimization-dynamics story here. -\n**The mechanism knows its regime.** Small offsets win at small scale: $\\pm$1–2 cells beat $\\pm$4 on CIFAR ($32 \\times 32$; nowhere to go), while ImageNet wanted $\\pm$4. And μδ is a**null result on CIFAR-100**(70.40% vs 70.39%) but**+1.2 on ImageNet**: on a $16 \\times 16$ grid the fixed field already covers everything worth covering; at $224^2$ there's room to move. Content-conditioned geometry pays exactly then.\n\n## Party Tricks\n\nTwo freebies that made me grin:\n\n**Zero-shot resolution transfer.** The field lives on *coordinates*, not token\nindices, so you train at $224^2$ (196 tokens) and just... evaluate bigger. Only\nthe tiny position embedding needs a bilinear resize; the Gaussians re-evaluate\non the new grid. On the d384 champion, $288^2$ scores **73.7% — higher than the\n72.0% at train resolution** — with zero fine-tuning. Still 71.6% at $384^2$; at\n$512^2$ the $\\pm$4-cell offset bound finally runs out of reach. Note for v2.\n\nLeft: latency vs tokens at d384/L12/h6 (batch 64, bf16; linear axes). Right: zero-shot transfer of the d384 factorized +μδσλ checkpoint — peak 73.7% at 288², then the offset bound bites.\n\n**Speed that scales.** Never building $N \\times N$ pays rent. Geometry replaces the\nsimilarity matrix, so SSOG skips QK entirely: SDPA pays $4d^2$ per layer for Q,\nK, V, and the output proj; SSOG pays $2d^2$, plus a few percent for steering.\nAt d256 that is ~1.0 GFLOPs/forward vs SDPA's ~1.5; at d384, ~4.4 vs ~6.3. From\n196 → 1024 tokens on the bigger twin, SDPA slows ~12× while axial fixed slows\n~7× and factorized +μδ sits in between (per-query maps aren't free). The point\nstands: **no $N^2$ anywhere**. Margins only grow with scale.\n\n## Opening the Hood\n\nFavorite part. We'll open the d384 champion. \"What did attention learn?\" usually\ngets a heatmap and a shrug. Here every head is a few Gaussians, so I can just\n*show* you the whole geometry — twelve layers, six heads:\n\nλ-weighted mixture density over relative position (per panel normalized). ★ = query (self); + = atom centers (brighter = larger λ).\n\nLayer 0 is a convolution in disguise: tight blobs around the query. Middle layers stretch into long vertical and horizontal bars — edge and strip detectors that span the image in one direction. Late layers go broad and global. All of it from random Gaussians plus gradient descent; nobody told layer 6 to become a line detector.\n\nAnd the gates? Every content tap starts frozen at $\\approx$ 0. Here's how far each layer opened them:\n\nLearned gate scales after training. All three started at ≈ 0 (dashed).\n\nEvery layer, every gate: opened. Not equally. $s(\\lambda)$ climbs hardest —\nlate layers love re-weighting atoms — while $s(\\mu\\delta)$ stays modest and\nfairly flat. The model has opinions about *which* content taps matter.\n\nThis is the figure I'd frame on my wall. Same bird as the playground, but\ninstead of another pretty heatmap I asked a bookkeeping question: at each\nlayer, how much does each patch **pull in** from the others, and how much is\n**pulled out** of it? With the residual-mixed mean-head matrix $\\hat{A}$:\n\n$$\\begin{aligned} \\mathrm{pull}_{\\mathrm{in},i} &= 1 - \\hat{A}_{ii}, \\\\ \\mathrm{pull}_{\\mathrm{out},i} &= \\sum_j \\hat{A}_{ji} - \\hat{A}_{ii}. \\end{aligned}$$\n\nNet export is their difference. Red = source (others read it more than it reads them); blue = gatherer. Sum over layers and you get the information economy of the whole forward pass:\n\nTop: input. Middle: per-layer net export (RdBu, 0-centered). Bottom: accumulated over all 12 layers — range about [−2.4, 7.5], ~38% of patches net exporters.\n\nEarly layers are almost balanced; everyone still mostly talks to themselves. Then the split opens: bird and post go red, background goes blue. By the last layers the silhouette is unmistakable, and the accumulated map is blunt about it. Only ~38% of patches are net exporters, and they are exactly the ones that matter. Everyone else is shopping there.\n\nThat's the point of steering, as an accounting identity: content doesn't have\nto *score* against content. It only has to decide who gets to be the source.\nThe field parked its exports on the subject. The background learned to shop.\n\n## Is This New?\n\nHonest answer — I've been on the wrong side of this question: the ingredients\nare all published. Content-free learned attention is\n[Synthesizer](https://arxiv.org/abs/2005.00743) (2020). Position-only\ninteraction is [AFT](https://arxiv.org/abs/2105.14103) and\n[LambdaNetworks](https://arxiv.org/abs/2102.08602). Gaussians *near* attention\nare everywhere: positional biases, RBF kernels over embeddings\n([guilty](../../posts/scaled-rbf-attention/); [HALO](../../posts/halo/) aims the same\ngeometry at classification heads), fixed weights for point clouds. Deforming\nattention with predicted offsets is\n[Deformable-DETR](https://arxiv.org/abs/2010.04159) /\n[DAT](https://arxiv.org/abs/2201.00520) territory.\n\nWhat I *haven't* found is the combo: a full attention operator that is (1) a\nGaussian mixture over relative position, (2) applied separably without ever\nforming $N \\times N$, and (3) deformed per-query through bounded, cold-started\nresiduals on μ, σ, and λ. DAT keeps content scores and samples discrete points;\nthis deforms a continuous field and never scores. Seen it published? Inbox is\nopen. Prior-art diligence is a team sport.\n\n## Would I Bet on It?\n\nCautiously, yes. Honest limits: still small models (3M and 12M), one recipe, vision only, and the μδ maps add a memory term I'd like to engineer away. The real proof is bigger models on bigger data — that job I leave to the GPU-upper class; I'm GPU-poor over here. Language is a different beast; content scoring is probably load-bearing there, though I'd love to be wrong.\n\nThe core finding still feels robust, and honestly surprising: **on images,\ncontent doesn't need to score. It only needs to steer.** Fixed geometry plus\ntiny bounded nudges matches SDPA on ImageNet, demolishes it on small data,\nscales near-linearly, transfers across resolutions for free, and turns \"what\ndid the model learn?\" into a question you answer with a ruler.\n\n## The Code\n\nThe clean, minimal implementation is on GitHub: `ssog/attention.py`\n\nholds the\nGaussian field and its SDPA twin, `ssog/vit.py`\n\nwires up a small ViT\nthat runs with either, and `examples/train_cifar100.py`\n\ntrains both so you can\ncompare them on your own GPU. Swapping attention mechanisms is a one-liner:\n\n```\nmodel = ViT(num_classes=100, attn=\"ssog\")   # the Gaussian field\nmodel = ViT(num_classes=100, attn=\"dot\")    # its SDPA twin\n```\n\nIf you poke at the mechanism, break it on your own data, or prove me wrong about language models, I'd love to hear what you find. And a star on the repo is always appreciated.", "url": "https://wpnews.pro/news/ssog-near-linear-visual-attention-that-doesn-t-score-but-steers", "canonical_source": "https://www.pisoni.ai/posts/ssog/", "published_at": "2026-08-16 09:30:59+00:00", "updated_at": "2026-08-16 09:40:40.298839+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "computer-vision", "neural-networks"], "entities": ["SSOG", "ImageNet", "SDPA"], "alternates": {"html": "https://wpnews.pro/news/ssog-near-linear-visual-attention-that-doesn-t-score-but-steers", "markdown": "https://wpnews.pro/news/ssog-near-linear-visual-attention-that-doesn-t-score-but-steers.md", "text": "https://wpnews.pro/news/ssog-near-linear-visual-attention-that-doesn-t-score-but-steers.txt", "jsonld": "https://wpnews.pro/news/ssog-near-linear-visual-attention-that-doesn-t-score-but-steers.jsonld"}}