Beyond Two Bytes per Letter: Tokenization Overhead in Cyrillic AI Systems

A new arXiv study (2608.21384v1) finds that Ukrainian and other Cyrillic-script languages face 68-121% token overhead on modern tokenizers and 220% on the older cl100k, compared to English, across nine production tokenizers and 8.37 million word forms. The researchers propose two mitigations: LLMLingua-2 reduces Ukrainian input length by 47-49% on an e-commerce RAG benchmark, and a balanced byte-level BPE tokenizer with a 200K vocabulary cap cuts the held-out UK/EN ratio from 2.22x to 1.30x. The findings suggest training data allocation contributes to Cyrillic tokenization overhead, with mitigation possible at both inference and tokenizer-design stages.

arXiv:2608.21384v1 Announce Type: new Abstract: Modern multilingual tokenizers often fragment Ukrainian and other underrepresented Cyrillic-script languages more heavily than English, creating disparities in cost and context capacity. We quantify this overhead across nine production tokenizers and five languages with standardized Cyrillic and Latin representations, covering 8.37 million word forms. On a corpus benchmark, Ukrainian shows 68-121% token overhead on modern tokenizers and 220% on the older cl100k, measured through full-text fertility on the BrUK and Brown corpora. Overhead is negatively associated with Cyrillic vocabulary allocation in the subset with independently verified English baselines, although the association is not statistically significant Spearman rho = -0.536, p = 0.215, n = 7 . We evaluate two mitigation strategies. LLMLingua-2 reduces Ukrainian input length by 47-49% on an e-commerce RAG benchmark of 1,536 products and 145 queries, with no compression-induced value losses among 80 retrievable cases. A balanced byte-level BPE tokenizer trained with a 200K vocabulary cap, converging at 158,184 actual entries, reduces the held-out UK/EN ratio from 2.22x to 1.30x. Romanization increases Ukrainian token counts by 2-19% on most tokenizers. Across the five languages, tokenization efficiency favors the script more prevalent in web data. These findings indicate that training data allocation contributes to Cyrillic tokenization overhead and that mitigation is possible at both inference and tokenizer-design stages.