AI Capability signal: Distilled 1B-parameter byte models eventually beat token models of the same size. Token-1B wins in the low-FLOP regime then plateaus; byt...

A September 11, 2026 arXiv paper (2609.12303) reports that distilled byte-level transformer models with roughly 1 billion parameters eventually surpass token-based models of the same size, reaching a higher downstream performance ceiling despite starting worse in the low-FLOP regime. The study, which swept layer-parameter-matched 1B models up to 1 trillion bytes of data across eight benchmarks, predicts distilled End-Of-Token-1B asymptotically beats distilled Token-1B by up to 4% and matches Token-1B performance using only one-sixth of the training data. The byte models' 256-byte vocabulary also cuts logit storage costs to roughly one-fifth and avoids top-k truncation, and the authors project their distilled End-Of-Token-1B models asymptotically exceed Llama 3.2-1B, Gemma-3-1B-pt, and Gemma 2B by up to 6.5%, 8.1%, and 2.1% respectively on averaged downstream tasks.

Computer Science Computation and Language Submitted on 11 Sep 2026 Title:Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models View PDF https://arxiv.org/pdf/2609.12303 HTML experimental https://arxiv.org/html/2609.12303v1 Abstract:Small models are made more capable through distillation from a larger one that shares their tokenization scheme. However, do distilled byte and token models behave similarly in terms of scaling trends as compute and data increases? To enable this comparison, we introduce two variants to efficiently convert token logits to Byte Logits: 1 approximate: Marginalize-It, and 2 exact: End-Of-Token. We then present the first large scale study of overtraining decoder-only dense transformer models varying two dimensions simultaneously: the tokenization scheme Tokens, Bytes, Bytes w/ eot and the training objective Distillation vs. Cross-Entropy , sweeping layer-parameter-matched models with roughly 1 billion parameters up to 1 trillion bytes of data. Across eight benchmarks spanning three categories: Multiple Choice QA, Language Generation, and Machine Translation, we find that Token-1B models outperform byte models End-Of-Token-1B and Bytes-1B in the low-FLOP regime but eventually plateau; byte models start worse yet surpass Token-1B models with more compute, reaching a higher downstream task performance ceiling. Extrapolating the average top-1 error vs. validation BPB scaling laws predicts that, asymptotically, distilled End-Of-Token-1B outperforms distilled Token-1B by up to 4%. They are also far more data efficient, matching the performance of distilled Token-1B using only one-sixth of the training data. Moreover, by operating over a small vocabulary of 256 bytes instead of on the order of 100K tokens, they circumvent the need for top-k truncation during logit dumping, while also reducing logit storage costs to roughly one-fifth. Finally, our downstream performance scaling laws predict that our distilled End-Of-Token-1B models asymptotically surpass the Llama 3.2-1B, Gemma-3-1B-pt, and Gemma 2B models on averaged downstream tasks by up to 6.5%, 8.1%, and 2.1%, respectively. Current browse context: cs.CL References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .