{"slug": "grouped-query-experts-mixture-of-experts-on-gqa-self-attention", "title": "Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention", "summary": "Vishesh Tripathi posted an arXiv paper on 18 June 2026 (revised 23 June 2026) proposing Grouped Query Experts (GQE), a mixture-of-experts layer built on grouped-query attention (GQA) in which a router selects k query-head experts per token while all key-value heads stay dense. At a 250M parameter scale and a fixed 30B token budget, GQE matched the all-active GQA baseline in downstream accuracy while activating half the query heads per token, preserving GQA's KV cache benefits and cutting only active query-head computation.", "body_md": "# Computer Science > Machine Learning\n\n  [Submitted on 18 Jun 2026 (\n\n[v1](https://arxiv.org/abs/2606.20945v1)), last revised 23 Jun 2026 (this version, v2)]\n# Title:Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention\n\n[View PDF](https://arxiv.org/pdf/2606.20945)\n\n[HTML (experimental)](https://arxiv.org/html/2606.20945v2)\n\nAbstract:Self-attention is central to Transformer performance and is often the most expensive part of the Transformer at long context lengths because its pairwise token interactions scale quadratically with sequence length. Standard dense attention also applies the same set of attention heads to every token regardless of token difficulty or information content. This uniform activation can waste compute, especially as sequences grow longer and attention cost increases rapidly. We propose Grouped Query Experts (GQE), a mixture-of-experts layer on top of grouped-query attention (GQA). Within each GQA group, a router selects k query-head experts per token while all key-value (KV) heads remain dense and unchanged. Thus, GQE keeps the KV cache benefits of GQA and reduces only the active query-head computation. On a fixed 30B token budget at the 250M parameter scale, GQE matches the all-active GQA baseline in downstream accuracy while activating half the query heads per token.\n    \n\n## Submission history\n\nFrom: Vishesh Tripathi [\n[view email](https://arxiv.org/show-email/4d6af19d/2606.20945)]\n\n**Thu, 18 Jun 2026 21:18:42 UTC (3,669 KB)**\n\n[\\[v1\\]](https://arxiv.org/abs/2606.20945v1)\n**[v2]** Tue, 23 Jun 2026 11:48:31 UTC (3,669 KB)\n\n### Additional Features\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\nIArxiv Recommender\n\n*(*[What is IArxiv?](https://iarxiv.org/about))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/grouped-query-experts-mixture-of-experts-on-gqa-self-attention", "canonical_source": "https://arxiv.org/abs/2606.20945", "published_at": "2026-11-03 11:31:10+00:00", "updated_at": "2026-11-03 11:51:45.965959+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "artificial-intelligence"], "entities": ["Vishesh Tripathi", "arXiv", "Grouped Query Experts (GQE)", "grouped-query attention (GQA)"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/grouped-query-experts-mixture-of-experts-on-gqa-self-attention", "markdown": "https://wpnews.pro/news/grouped-query-experts-mixture-of-experts-on-gqa-self-attention.md", "text": "https://wpnews.pro/news/grouped-query-experts-mixture-of-experts-on-gqa-self-attention.txt", "jsonld": "https://wpnews.pro/news/grouped-query-experts-mixture-of-experts-on-gqa-self-attention.jsonld"}}