{"slug": "can-this-geometry-kill-ai-attention", "title": "Can This Geometry Kill AI Attention?", "summary": "An independent reproduction of Zhang Chong's December 2025 paper \"Attention Is Not What You Need\" found a 22.6% performance gap versus the paper's claim that its \"Grassmann Flows\" architecture lands within 10-15% of size-matched Transformers on perplexity, according to reproduction author Elliot Arledge. The architecture replaces the L-by-L attention matrix with fixed-offset neighbor pairings (offsets 1, 2, 4, 8, 12 and 16) encoded as 496 Plücker coordinates on a 32-dimensional Grassmann manifold, scaling linearly with sequence length. The paper, tested only on 13M-18M-parameter models on Wikitext-2 with no released code, also pairs each token with future tokens (t + Δ), a formulation that could allow data leakage and compromise causal evaluation.", "body_md": "## Attention’s quadratic bill—and a radical escape route\n\nToday’s large language models, from GPT to Gemini, rely on the Transformer architecture’s **attention mechanism**. This system allows every token in a sequence to compare itself to every other token, creating an L-by-L matrix of interactions. While powerful, this global comparison incurs a computational and memory cost that grows quadratically with sequence length, posing a significant scaling challenge.\n\nA recent paper from December 2025, “Attention Is Not What You Need” by Zhang Chong, proposes a radical departure. Instead of the costly L-by-L attention matrix, Chong’s \"Grassmann Flows\" architecture replaces it with a mechanism rooted in 19th-century differential geometry. This isn't an established alternative, but a novel experimental concept.\n\nThe core idea involves shrinking token representations from 256 to 32 dimensions and pairing them with a fixed set of neighbors at specific distances (1, 2, 4, 8, 12, and 16 positions). Each pair forms a plane within a 32-dimensional **Grassmann manifold**, encoded by 496 Plücker coordinates. A learned gate then mixes this geometric information into the token’s original representation.\n\nBecause each token interacts only with a fixed number of neighbors, this approach scales linearly with sequence length, dramatically reducing complexity. Early results on small models (13M to 18M parameters) show perplexity within 10-15% of size-matched Transformers, and even a slight edge on one classification task. However, lower computational complexity alone does not guarantee useful language model performance; the true test lies in scaling and independent reproducibility.\n\n## The 19th-century geometry hiding in each token pair\n\nInstead of a full attention matrix, this new architecture compresses each token’s 256-dimensional state to just 32 dimensions. It then pairs each token with a select set of neighbors at fixed offsets: 1, 2, 4, 8, 12, and 16 positions. This local, multi-scale interaction avoids the quadratic cost of global attention.\n\nTwo 32-dimensional vectors, representing a token pair, define a plane within a 32-dimensional space. In mathematics, the collection of all such planes forms a **Grassmann manifold**. The model encodes each of these planes using **Plücker coordinates**, generating 496 distinct numerical features for every pair.\n\nA learned gate then dictates how much of this rich 496-dimensional geometric information integrates into the token’s original state. This mechanism offers a linear scaling cost with sequence length. The paper claims this geometrical approach allows for mathematically inspectable representations, offering a potential avenue for **interpretability**—a compelling research angle, though not yet a proven benefit.\n\n## Promising numbers meet an ugly reproducibility test\n\nInitial results from the paper, “[Attention Is Not What You Need](https://arxiv.org/abs/2512.19428),” seemed promising. The author tested models with roughly 13–18 million parameters on Wikitext-2, claiming perplexity within 10–15% of comparable Transformers. For a classification task, a DistilBERT head using this geometric approach achieved 85.38% accuracy, slightly outperforming a Transformer baseline at 85.11%.\n\nHowever, independent reproducibility efforts have raised questions. One reproduction by Elliot Arledge reported a 22.6% performance gap, significantly wider than the paper’s claims. While a second attempt yielded closer numbers, the discrepancy suggests that undocumented implementation choices may critically influence performance.\n\nTwo central objections persist. First, the author provided no code, making it impossible to verify the implementation details directly. Second, a key formula in the paper describes pairing each token with future tokens ($t + \\Delta$), which could compromise **causal evaluation** by allowing the model to \"peek\" at upcoming information during training. If applied as described, this **data leakage** would invalidate the reported performance metrics for autoregressive tasks.\n\nEnjoying this? Get one like it in your inbox each morning.\n\none email a day · unsubscribe in two clicks · no third-party tracking\n\n## A research lead, not an attention replacement\n\nEvidence from models of 13–18 million parameters on Wikitext-2, even with reported perplexity within 10–15% of Transformers and a slight edge on DistilBERT classification, does not establish viability at scale. These are **toy models** by current standards, and their performance on small benchmarks cannot predict behavior in billion-parameter language models or long-context applications.\n\nThis geometric approach enters a crowded field of linear-time alternatives, including **Mamba**, RWKV, and RetNet. Grassmann flows must demonstrate superior quality and reproducibility, not merely linear complexity, to gain traction. The absence of public code, coupled with potential causal leakage in the paper's formulation and conflicting reproduction results—one reporting a 22.6% performance gap—raises significant concerns.\n\nUntil the research community has access to:\n\n- Publicly available, reproducible code\n- Rigorous causal tests to rule out data leakage\n- Independent replications on larger models and datasets\n\nThe \"Grassmann flow\" is best viewed as a promising research lead for exploring structured representations, particularly for interpretability. Broader claims for replacing attention require far more robust validation.\n\n## Frequently Asked Questions\n\n### What are Grassmann flows?\n\nThey are a proposed sequence-modeling architecture that represents pairs of token vectors as geometric subspaces and uses those features instead of standard self-attention.\n\n### How could Grassmann flows reduce computation?\n\nThey pair each token with a fixed set of neighbors rather than comparing every token with every other token, aiming for linear rather than quadratic sequence-length scaling.\n\n### Has Grassmann flows been shown to outperform Transformers?\n\nNo. The paper reports small-model results near, but generally below, size-matched Transformers; independent reproductions have reported differing performance.\n\n### Why are researchers questioning the paper’s results?\n\nThe paper lacks reference code, implementation details are disputed, and commenters have raised concerns about future-token indexing in causal language modeling.", "url": "https://wpnews.pro/news/can-this-geometry-kill-ai-attention", "canonical_source": "https://www.stork.ai/blog/can-this-geometry-kill-ai-attention", "published_at": "2026-10-30 11:37:13+00:00", "updated_at": "2026-10-30 12:21:38.108727+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "natural-language-processing"], "entities": ["Zhang Chong", "Elliot Arledge", "Grassmann Flows", "Attention Is Not What You Need", "Wikitext-2", "DistilBERT", "Transformer", "Grassmann manifold"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/can-this-geometry-kill-ai-attention", "markdown": "https://wpnews.pro/news/can-this-geometry-kill-ai-attention.md", "text": "https://wpnews.pro/news/can-this-geometry-kill-ai-attention.txt", "jsonld": "https://wpnews.pro/news/can-this-geometry-kill-ai-attention.jsonld"}}