{"slug": "tapered-language-models", "title": "Tapered Language Models", "summary": "Researchers introduced Tapered Language Models (TLMs), an architectural principle that allocates more parameter capacity to earlier layers and less to later layers under a fixed budget. Across four architectures and three model scales, tapering MLP width via a cosine schedule consistently improved perplexity and downstream performance over uniform baselines at no additional cost. The findings establish depth-aware capacity allocation as a simple, architecture-agnostic axis of language model design.", "body_md": "# Computer Science > Machine Learning\n\n[Submitted on 22 Jun 2026]\n\n# Title:Tapered Language Models\n\n[View PDF](/pdf/2606.23670)\n\n[HTML (experimental)](https://arxiv.org/html/2606.23670v1)\n\nAbstract:Modern language models, including transformer, recurrent, and memory-based variants, share a common chassis: a stack of identical layers in which parameters are allocated uniformly across depth. This is a default inherited from the original transformer and largely unchanged since, yet a growing body of evidence suggests that layers contribute non-uniformly to the final output, with later layers refining the residual stream rather than transforming it. We ask whether parameter capacity should reflect this asymmetry. Our controlled experiment shows that, under a fixed budget, allocating more capacity to earlier layers and less to later layers improves perplexity over a uniform-width baseline, while the reverse allocation hurts. Building on this result, we introduce Tapered Language Models (TLMs), an architectural principle in which a parameter-bearing component is monotonically tapered across depth under a fixed total budget. MLPs are the natural site for this instantiation: they dominate parameter count across all modern LM families and expose width as a single, clean axis of variation. Across three model scales and four architectures (Transformer, Gated Attention, Hope-attention, and Titans), tapering MLP width via a smooth cosine schedule consistently improves perplexity and downstream benchmark performance over uniform baselines, at no additional parameter or compute cost. These findings establish depth-aware capacity allocation as a simple, architecture-agnostic axis of language model design, a free lever hidden in plain sight.\n\n### Current browse context:\n\ncs.LG\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer\n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers\n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps\n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations\n\n*(*[What are Smart Citations?](https://www.scite.ai/))# Code, Data and Media Associated with this Article\n\nalphaXiv\n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers\n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub\n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub\n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face\n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast\n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower\n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender\n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\nIArxiv Recommender\n\n*(*[What is IArxiv?](https://iarxiv.org/about))# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [ Learn more about arXivLabs](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/tapered-language-models", "canonical_source": "https://arxiv.org/abs/2606.23670", "published_at": "2026-06-27 11:56:00+00:00", "updated_at": "2026-06-27 12:04:50.173876+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "neural-networks", "ai-research"], "entities": ["Transformer", "Gated Attention", "Hope-attention", "Titans"], "alternates": {"html": "https://wpnews.pro/news/tapered-language-models", "markdown": "https://wpnews.pro/news/tapered-language-models.md", "text": "https://wpnews.pro/news/tapered-language-models.txt", "jsonld": "https://wpnews.pro/news/tapered-language-models.jsonld"}}