04:00
2026-08-19
arxiv.org
natural-language-processing
What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?
A new arXiv study (2608.17325v1) finds that jointly optimizing tokenization with language modeling fundamentally alters token structure, with SSLMs recovering morphologically aligned and contextually …