# Small pre-tokenization bugs with a big multilingual price > Source: > Published: 2026-09-01 20:55:34+00:00 In [the first post in](https://tokencontributions.substack.com/p/a-short-introduction-to-pre-tokenization) a series on pre-tokenization, I showed the regex GPT-2 uses to split text before BPE. Its word branch includes only `\p{L}`, Unicode’s Letter category. It leaves out `\p{M}`, the Mark category, which contains combining accents, tone marks, and many other ordinary parts of written words. The omission first appeared in GPT-2. It survived several generations of OpenAI models and still affects a surprising number of current models, most notably GLM-5. Claude does not inherit this regex, but has its own problem with the same characters. ## What GPT-2 does Consider these three words: ``` ọ̀rọ̀ Yoruba: word, speech, language (5 code points, 2 Marks) राष्ट्रीय Hindi: national (9 code points, 4 Marks) วันนี้ Thai: today (6 code points, 3 Marks) ``` The two grave accents in the Yoruba word belong to Unicode’s Mark category. So do the two vowel signs and two [viramas](https://en.wikipedia.org/wiki/Virama) in Hindi, and the two [vowel signs](https://en.wikipedia.org/wiki/Thai_script#Vowels) and [tone mark](https://en.wikipedia.org/wiki/Thai_script#Tone) in Thai. These are ordinary parts of the words, not punctuation. GPT-2’s regex sees letters and marks as different kinds of text and splits them in pre-tokenization. This is a hard boundary. BPE runs on each resulting chunk separately, so it can never learn a token spanning one of these cuts, regardless of how common some of these words may be. ``` ọ | ̀ | rọ | ̀ र | ा | ष | ् | ट | ् | र | ी | य ว | ั | นน | ี้ ``` ## GPT-4 retained the bug This mattered less for GPT-2, an English-focused model. Accented letters used in English also tend to have precomposed Unicode forms, as in `café`. However, GPT-4 retains the issue, and many more models inherit from it. Specifically, the relevant word alternative became `[^\r\n\p{L}\p{N}]?\p{L}+` This allows chunks such as `_data` and `\data`. It also changes the appearance of the mark bug: since a mark is neither a letter nor a number, the optional prefix attaches it to the letters that follow. This produces fewer chunks, making the bug slightly less disastrous: ``` ọ | ̀rọ | ̀ र | ाष | ्ट | ्र | ीय ว | ันน | ี้ ``` OpenAI finally fixed this in `o200k`, introduced with GPT-4o and still used by GPT-5.6. Its word branches include `\p{M}`. ## Who still has the bug? The bug now has two main lines of descent: the original GPT-2 regex[1](#footnote-1) and the cl100k-shaped version introduced with GPT-4. Most notable among these are Llama 3, Qwen 3 and GLM-4/5. Several newer tokenizers have fixed the mark handling or never inherited the bug, including Llama 4, Qwen 3.5+, Kimi K3, DeepSeek V3/V4, and recent Mistral models. These tokenizers all include marks in their word alternatives. Gemini and Gemma use SentencePiece and do not use this regex pre-tokenization step. Among current flagship models, GLM-5 is probably the most prominent holdout. ## What about Claude? Claude has a different issue with combining marks. Remember the [boundary markers on words](https://tokencontributions.substack.com/i/206706614/boundary-markers-on-words)? It turns out spans are marked if they have the Unicode’s derived *Alphabetic* property[2](#footnote-2). The catch is that *Alphabetic* includes only some characters in Unicode’s Mark category. Ordinary combining accents, viramas, and tone marks fall outside their words, while marks such as Thai vowel signs remain inside. ``` ọ̀rọ̀ → ^ọ$ ̀ ^rọ$ ̀ (interrupted at its combining graves) राष्ट्रीय → ^राष$ ् ^ट$ ् ^रीय$ (interrupted at viramas) วันนี้ → ^วันนี$ ้ (only the final tone mark split off) ``` Note that these are marked spans before vocabulary encoding, not the final tokens. ## Multilingual fairness is hard. Fixing these bugs isn’t. Choosing the multilingual balance of a tokenizer is genuinely difficult. Vocabulary space is limited, and giving more tokens to one language leaves fewer for another. But this problem happens before any of those trade-offs. Splitting a word at its combining marks prevents the tokenizer from learning even the most common words in some languages. On our parallel sample, Qwen 3 uses 4.4 times as many tokens per character for Hindi as for English. In Qwen 3.5, which fixed the regex, that falls to 2 times. This affects hundreds of millions of people. They get less text in the same context window, and pay more for the same amount of text, partly because ordinary spelling marks are mistaken for word boundaries. [1](#footnote-anchor-1) Few prominent models still use the GPT-2 regex. Cohere used it for Command R, R+, and Command A, then switched Command A+ to an o200k-based tokenizer. It remains the built-in regex used by Hugging Face Tokenizers’ `ByteLevel` pre-tokenizer when `use_regex=True`. [2](#footnote-anchor-2) More precisely, this path is limited to *Alphabetic* characters in Unicode’s main 65,536-code-point range, excluding Han and Hangul.