04:00
2026-09-30
arxiv.org
natural-language-processing
The Price of Token Boundaries: Compression Certificates and Prediction
A new arXiv paper (2609.35869v1) measures the compression cost of pre-tokenisation boundary rules, finding that on English Wikipedia such boundaries increase the optimal token count by 28.3–36.8%. The…