{"slug": "attention-is-not-what-you-need", "title": "Attention Is Not What You Need", "summary": "A paper submitted to arXiv on 22 Dec 2025 proposes an attention-free architecture, the Causal Grassmann layer, that replaces the L by L attention matrix with token reduction, encoding of local token pairs as two-dimensional subspaces on a Grassmann manifold via Plucker coordinates, and gated mixing. On Wikitext-2, purely Grassmann-based models with 13 to 18 million parameters reach validation perplexities within about 10 to 15 percent of size-matched Transformers, and on SNLI a Grassmann-Plucker head on DistilBERT slightly beats a Transformer head at 0.8550 validation and 0.8538 test accuracy versus 0.8545 and 0.8511. The authors show linear scaling in sequence length for fixed rank and argue manifold-based designs offer a more structured route to invariant-based interpretations of neural reasoning.", "body_md": "# Computer Science > Machine Learning\n\n  [Submitted on 22 Dec 2025]\n\n# Title:Attention Is Not What You Need\n\n[View PDF](https://arxiv.org/pdf/2512.19428)\n\n[HTML (experimental)](https://arxiv.org/html/2512.19428v1)\n\nAbstract:We revisit a basic question in sequence modeling: is explicit self-attention actually necessary for strong performance and reasoning? We argue that standard multi-head attention is best seen as a form of tensor lifting: hidden vectors are mapped into a high-dimensional space of pairwise interactions, and learning proceeds by constraining this lifted tensor through gradient descent. This mechanism is extremely expressive but mathematically opaque, because after many layers it becomes very hard to describe the model with a small family of explicit invariants.\n\nTo explore an alternative, we propose an attention-free architecture based on Grassmann flows. Instead of forming an L by L attention matrix, our Causal Grassmann layer (i) linearly reduces token states, (ii) encodes local token pairs as two-dimensional subspaces on a Grassmann manifold via Plucker coordinates, and (iii) fuses these geometric features back into the hidden states through gated mixing. Information therefore propagates by controlled deformations of low-rank subspaces over multi-scale local windows, so the core computation lives on a finite-dimensional manifold rather than in an unstructured tensor space.\n\nOn the Wikitext-2 language modeling benchmark, purely Grassmann-based models with 13 to 18 million parameters achieve validation perplexities within about 10 to 15 percent of size-matched Transformers. On the SNLI natural language inference task, a Grassmann-Plucker head on top of DistilBERT slightly outperforms a Transformer head, with best validation and test accuracies of 0.8550 and 0.8538 compared to 0.8545 and 0.8511. We analyze the complexity of Grassmann mixing, show linear scaling in sequence length for fixed rank, and argue that such manifold-based designs offer a more structured route toward geometric and invariant-based interpretations of neural reasoning.\n\n### Additional Features\n\n### Current browse context:\n\ncs.LG\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\nIArxiv Recommender\n\n*(*[What is IArxiv?](https://iarxiv.org/about))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/attention-is-not-what-you-need", "canonical_source": "https://arxiv.org/abs/2512.19428", "published_at": "2026-10-25 23:52:05+00:00", "updated_at": "2026-10-26 00:13:32.889899+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "natural-language-processing", "neural-networks", "large-language-models"], "entities": ["arXiv", "Causal Grassmann layer", "Grassmann manifold", "Plucker coordinates", "Wikitext-2", "SNLI", "DistilBERT"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/attention-is-not-what-you-need", "markdown": "https://wpnews.pro/news/attention-is-not-what-you-need.md", "text": "https://wpnews.pro/news/attention-is-not-what-you-need.txt", "jsonld": "https://wpnews.pro/news/attention-is-not-what-you-need.jsonld"}}