{"slug": "ai-capability-signal-distilled-1b-parameter-byte-models-eventually-beat-token-of", "title": "AI Capability signal: Distilled 1B-parameter byte models eventually beat token models of the same size. Token-1B wins in the low-FLOP regime then plateaus; byt...", "summary": "A September 11, 2026 arXiv paper (2609.12303) reports that distilled byte-level transformer models with roughly 1 billion parameters eventually surpass token-based models of the same size, reaching a higher downstream performance ceiling despite starting worse in the low-FLOP regime. The study, which swept layer-parameter-matched 1B models up to 1 trillion bytes of data across eight benchmarks, predicts distilled End-Of-Token-1B asymptotically beats distilled Token-1B by up to 4% and matches Token-1B performance using only one-sixth of the training data. The byte models' 256-byte vocabulary also cuts logit storage costs to roughly one-fifth and avoids top-k truncation, and the authors project their distilled End-Of-Token-1B models asymptotically exceed Llama 3.2-1B, Gemma-3-1B-pt, and Gemma 2B by up to 6.5%, 8.1%, and 2.1% respectively on averaged downstream tasks.", "body_md": "# Computer Science > Computation and Language\n\n  [Submitted on 11 Sep 2026]\n\n# Title:Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models\n\n[View PDF](https://arxiv.org/pdf/2609.12303)\n\n[HTML (experimental)](https://arxiv.org/html/2609.12303v1)\n\nAbstract:Small models are made more capable through distillation from a larger one that shares their tokenization scheme. However, do distilled byte and token models behave similarly in terms of scaling trends as compute and data increases? To enable this comparison, we introduce two variants to efficiently convert token logits to Byte Logits: 1) approximate: Marginalize-It, and 2) exact: End-Of-Token. We then present the first large scale study of overtraining decoder-only dense transformer models varying two dimensions simultaneously: the tokenization scheme (Tokens, Bytes, Bytes w/ eot) and the training objective (Distillation vs. Cross-Entropy), sweeping layer-parameter-matched models with roughly 1 billion parameters up to 1 trillion bytes of data. Across eight benchmarks spanning three categories: Multiple Choice QA, Language Generation, and Machine Translation, we find that Token-1B models outperform byte models (End-Of-Token-1B and Bytes-1B) in the low-FLOP regime but eventually plateau; byte models start worse yet surpass Token-1B models with more compute, reaching a higher downstream task performance ceiling. Extrapolating the average top-1 error vs. validation BPB scaling laws predicts that, asymptotically, distilled End-Of-Token-1B outperforms distilled Token-1B by up to 4%. They are also far more data efficient, matching the performance of distilled Token-1B using only one-sixth of the training data. Moreover, by operating over a small vocabulary of 256 bytes instead of on the order of 100K tokens, they circumvent the need for top-k truncation during logit dumping, while also reducing logit storage costs to roughly one-fifth. Finally, our downstream performance scaling laws predict that our distilled End-Of-Token-1B models asymptotically surpass the Llama 3.2-1B, Gemma-3-1B-pt, and Gemma 2B models on averaged downstream tasks by up to 6.5%, 8.1%, and 2.1%, respectively.\n    \n\n### Current browse context:\n\ncs.CL\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/ai-capability-signal-distilled-1b-parameter-byte-models-eventually-beat-token-of", "canonical_source": "https://arxiv.org/abs/2609.12303", "published_at": "2026-09-11 00:00:00+00:00", "updated_at": "2026-09-28 05:48:25.341517+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research", "natural-language-processing"], "entities": ["arXiv", "Llama 3.2-1B", "Gemma-3-1B-pt", "Gemma 2B"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai-capability-signal-distilled-1b-parameter-byte-models-eventually-beat-token-of", "markdown": "https://wpnews.pro/news/ai-capability-signal-distilled-1b-parameter-byte-models-eventually-beat-token-of.md", "text": "https://wpnews.pro/news/ai-capability-signal-distilled-1b-parameter-byte-models-eventually-beat-token-of.txt", "jsonld": "https://wpnews.pro/news/ai-capability-signal-distilled-1b-parameter-byte-models-eventually-beat-token-of.jsonld"}}