Merge-Based Vocabulary Extension Stalls at the Pre-Tokenizer: An English-Preserving Repair for Nepali, Tested in Continued Pretraining

A small edit to the regular expression used by Llama-3 and Qwen3 pre-tokenizers brings Nepali to 1.01 times the token count of equivalent English text, according to an arXiv paper (2610.33016v1) that releases the tokenizers and code. The deployed regex admits only Unicode letters, so Devanagari vowel signs split Nepali words and models spend 2.6 to 5.8 times as many tokens on Nepali as on English; merge-based vocabulary extension cannot fix this, with no number of new merges bringing the premium below about 2.2 for Llama-3 and Qwen3, and ten released extensions ending within 4% of that floor. With 32,000 new merges the repair cuts Nepali tokens by about 54% versus standard extension, though in small-budget continued pretraining of Llama-3.2-1B and Qwen3-0.6B the original tokenizer gave the lowest held-out Nepali bits per byte under every recipe tested, and whether the savings yield a favorable latency and quality trade-off in generation remains untested.

arXiv:2610.33016v1 Announce Type: new Abstract: Llama-3, Qwen3 and many other open LLMs pre-tokenize text with a regular expression whose word class admits only Unicode letters. Devanagari vowel signs are combining marks, so this regex splits Nepali words at every vowel sign, and models that use it spend 2.6 to 5.8 times as many tokens on Nepali as on the same English text. Merge-based vocabulary extension cannot join the split pieces: for Llama-3 and Qwen3 no number of new merges brings this premium below about 2.2, and ten released extensions already end within 4% of this pre-token floor. We give a small edit to the deployed regex that provably keeps every token id of text without combining marks or Devanagari characters. With 32,000 new merges it brings Nepali to 1.01 times English, about 54% fewer Nepali tokens than standard extension. In small-budget continued pretraining of Llama-3.2-1B and Qwen3-0.6B, the original tokenizer gave the lowest held-out Nepali bits per byte under every recipe we tested. Matching the repair's training steps to standard extension, with about 35% of the data repeated, reversed their order on Qwen but not on Llama. Whether the token savings give a favourable latency and quality trade-off in generation remains untested. We release the tokenizers and code.