{"slug": "byte-language-models-scaling-emergent-abstractions-and-information-allocation", "title": "Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation", "summary": "Byte Transformers trained with token-superposition training and hash embeddings consistently outperform subword Transformers as model size scales, according to an arXiv paper submitted on 5 October 2026 by researchers studying tokenizer-free language models. The authors found byte Transformers learn segmentation-like positions that act as external tokenizers, and that restricting up to 25% of intermediate layers to these local representations preserves downstream performance. Exploiting the non-uniform generation difficulty concentrated near local structure boundaries for speculative decoding yielded 3.4x more accepted tokens than in subword Transformers.", "body_md": "# Computer Science > Computation and Language\n\n  [Submitted on 5 Oct 2026]\n\n# Title:Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation\n\n[View PDF](https://arxiv.org/pdf/2610.05978)\n\n[HTML (experimental)](https://arxiv.org/html/2610.05978v1)\n\nAbstract:Tokenizer-free language models remove the inductive bias of fixed tokenizers by modeling text directly as bytes, but the resulting longer sequences substantially increase computation and eliminate explicit text abstractions. We ask whether this additional computation can be useful, and whether standard Transformers can learn the abstractions that tokenization provides. We study these questions on Transformers without specialized tokenization-related architectures. With token-superposition training and hash embeddings, byte Transformers consistently outperform subword Transformers as model size scales. We further find that byte Transformers build local text abstractions as external tokenizers: a set of segmentation-like positions are used to collect local context representations, and restricting up to $25\\%$ of intermediate layers to these local representations preserves downstream performance. Finally, these learned structures induce highly non-uniform generation difficulty, with uncertainty concentrated near local structure boundaries; exploiting them for speculative decoding yields $3.4\\times$ more accepted tokens than in subword Transformers.\n    \n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/byte-language-models-scaling-emergent-abstractions-and-information-allocation", "canonical_source": "https://arxiv.org/abs/2610.05978", "published_at": "2026-10-12 13:31:57+00:00", "updated_at": "2026-10-12 13:47:08.275537+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "ai-research", "machine-learning", "artificial-intelligence"], "entities": ["arXiv", "Byte Transformers", "Transformers", "subword Transformers"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/byte-language-models-scaling-emergent-abstractions-and-information-allocation", "markdown": "https://wpnews.pro/news/byte-language-models-scaling-emergent-abstractions-and-information-allocation.md", "text": "https://wpnews.pro/news/byte-language-models-scaling-emergent-abstractions-and-information-allocation.txt", "jsonld": "https://wpnews.pro/news/byte-language-models-scaling-emergent-abstractions-and-information-allocation.jsonld"}}