{"slug": "small-pre-tokenization-bugs-with-a-big-multilingual-price", "title": "Small pre-tokenization bugs with a big multilingual price", "summary": "A developer's analysis of pre-tokenization regexes shows that GPT-2's word-splitting pattern omitted Unicode's Mark category, a bug inherited by GPT-4, Llama 3, Qwen 3, and GLM-4/5, forcing BPE to tokenize combining accents, viramas, and tone marks separately from their base letters. The writeup also finds Claude's boundary-marker scheme splits words at ordinary combining marks because Unicode's derived Alphabetic property excludes many Mark characters. OpenAI fixed the issue in the o200k tokenizer used by GPT-4o and GPT-5.6, while Llama 4, Qwen 3.5+, Kimi K3, DeepSeek V3/V4, and recent Mistral models also handle marks correctly.", "body_md": "In [the first post in](https://tokencontributions.substack.com/p/a-short-introduction-to-pre-tokenization) a series on pre-tokenization, I showed the regex GPT-2 uses to split text before BPE. Its word branch includes only `\\p{L}`, Unicode’s Letter category. It leaves out `\\p{M}`, the Mark category, which contains combining accents, tone marks, and many other ordinary parts of written words.\n\nThe omission first appeared in GPT-2. It survived several generations of OpenAI models and still affects a surprising number of current models, most notably GLM-5.\n\nClaude does not inherit this regex, but has its own problem with the same characters.\n\n## What GPT-2 does\n\nConsider these three words:\n\n```\nọ̀rọ̀       Yoruba: word, speech, language (5 code points, 2 Marks)\nराष्ट्रीय      Hindi: national                (9 code points, 4 Marks)\nวันนี้       Thai: today                    (6 code points, 3 Marks)\n```\n\nThe two grave accents in the Yoruba word belong to Unicode’s Mark category. So do the two vowel signs and two [viramas](https://en.wikipedia.org/wiki/Virama) in Hindi, and the two [vowel signs](https://en.wikipedia.org/wiki/Thai_script#Vowels) and [tone mark](https://en.wikipedia.org/wiki/Thai_script#Tone) in Thai. These are ordinary parts of the words, not punctuation.\n\nGPT-2’s regex sees letters and marks as different kinds of text and splits them in pre-tokenization. This is a hard boundary. BPE runs on each resulting chunk separately, so it can never learn a token spanning one of these cuts, regardless of how common some of these words may be.\n\n```\nọ | ̀ | rọ | ̀\nर | ा | ष | ् | ट | ् | र | ी | य\nว | ั | นน | ี้\n```\n\n## GPT-4 retained the bug\n\nThis mattered less for GPT-2, an English-focused model. Accented letters used in English also tend to have precomposed Unicode forms, as in `café`.\n\nHowever, GPT-4 retains the issue, and many more models inherit from it. Specifically, the relevant word alternative became `[^\\r\\n\\p{L}\\p{N}]?\\p{L}+`\n\nThis allows chunks such as `_data` and `\\data`. It also changes the appearance of the mark bug: since a mark is neither a letter nor a number, the optional prefix attaches it to the letters that follow. This produces fewer chunks, making the bug slightly less disastrous:\n\n```\nọ | ̀rọ | ̀\nर | ाष | ्ट | ्र | ीय\nว | ันน | ี้\n```\n\nOpenAI finally fixed this in `o200k`, introduced with GPT-4o and still used by GPT-5.6. Its word branches include `\\p{M}`.\n\n## Who still has the bug?\n\nThe bug now has two main lines of descent: the original GPT-2 regex[1](#footnote-1) and the cl100k-shaped version introduced with GPT-4. Most notable among these are Llama 3, Qwen 3 and GLM-4/5.\n\nSeveral newer tokenizers have fixed the mark handling or never inherited the bug, including Llama 4, Qwen 3.5+, Kimi K3, DeepSeek V3/V4, and recent Mistral models. These tokenizers all include marks in their word alternatives. Gemini and Gemma use SentencePiece and do not use this regex pre-tokenization step.\n\nAmong current flagship models, GLM-5 is probably the most prominent holdout.\n\n## What about Claude?\n\nClaude has a different issue with combining marks. Remember the [boundary markers on words](https://tokencontributions.substack.com/i/206706614/boundary-markers-on-words)? It turns out spans are marked if they have the Unicode’s derived *Alphabetic* property[2](#footnote-2).\n\nThe catch is that *Alphabetic* includes only some characters in Unicode’s Mark category. Ordinary combining accents, viramas, and tone marks fall outside their words, while marks such as Thai vowel signs remain inside.\n\n```\nọ̀rọ̀      →  ^ọ$      ̀   ^rọ$    ̀        (interrupted at its combining graves)\nराष्ट्रीय     →   ^राष$   ्   ^ट$     ्  ^रीय$  (interrupted at viramas)\nวันนี้      →  ^วันนี$    ้                   (only the final tone mark split off)\n```\n\nNote that these are marked spans before vocabulary encoding, not the final tokens.\n\n## Multilingual fairness is hard. Fixing these bugs isn’t.\n\nChoosing the multilingual balance of a tokenizer is genuinely difficult. Vocabulary space is limited, and giving more tokens to one language leaves fewer for another.\n\nBut this problem happens before any of those trade-offs. Splitting a word at its combining marks prevents the tokenizer from learning even the most common words in some languages.\n\nOn our parallel sample, Qwen 3 uses 4.4 times as many tokens per character for Hindi as for English. In Qwen 3.5, which fixed the regex, that falls to 2 times. This affects hundreds of millions of people. They get less text in the same context window, and pay more for the same amount of text, partly because ordinary spelling marks are mistaken for word boundaries.\n\n[1](#footnote-anchor-1)\n\nFew prominent models still use the GPT-2 regex. Cohere used it for Command R, R+, and Command A, then switched Command A+ to an o200k-based tokenizer. It remains the built-in regex used by Hugging Face Tokenizers’ `ByteLevel` pre-tokenizer when `use_regex=True`.\n\n[2](#footnote-anchor-2)\n\nMore precisely, this path is limited to *Alphabetic* characters in Unicode’s main 65,536-code-point range, excluding Han and Hangul.", "url": "https://wpnews.pro/news/small-pre-tokenization-bugs-with-a-big-multilingual-price", "canonical_source": "https://tokencontributions.substack.com/p/small-pre-tokenization-bugs-with", "published_at": "2026-09-01 20:55:34+00:00", "updated_at": "2026-09-14 02:24:52.191016+00:00", "lang": "en", "topics": ["natural-language-processing", "large-language-models", "ai-research", "ai-ethics"], "entities": ["OpenAI", "GPT-2", "GPT-4", "GPT-4o", "Claude", "GLM-5", "Llama 3", "Qwen 3"], "alternates": {"html": "https://wpnews.pro/news/small-pre-tokenization-bugs-with-a-big-multilingual-price", "markdown": "https://wpnews.pro/news/small-pre-tokenization-bugs-with-a-big-multilingual-price.md", "text": "https://wpnews.pro/news/small-pre-tokenization-bugs-with-a-big-multilingual-price.txt", "jsonld": "https://wpnews.pro/news/small-pre-tokenization-bugs-with-a-big-multilingual-price.jsonld"}}