Tokenizer fairness for Urdu: 5.39× more tokens on GPT-2, 2.72× on Qwen2.5

A pilot study of 30 parallel English–Urdu sentence pairs across 17 domains found that Urdu text consumes 5.39× more tokens than English on GPT-2, 3.73× more on Mistral v0.3, and 2.72× more on Qwen2.5. The author reports that vocabulary size alone does not explain the gap: GPT-2's ~50k vocabulary is larger than Mistral's ~32k, yet GPT-2 performs worse, suggesting training data matters at least as much. Per character, English averages ~5.35 characters per token on GPT-2 versus ~1.34 for Urdu; the author cautions that 30 sentences is a pilot rather than a precise multiplier and covers only formal register, and plans to extend the corpus to Pashto, Sindhi and Shahmukhi Punjabi.

Hi all — I measured how many tokens Urdu costs compared with English for the same meaning, and wanted to share the numbers and ask for feedback. I built a small parallel corpus 30 sentence pairs, 17 domains, CC-BY-4.0 and tokenized both sides with three tokenizers: | Tokenizer | Vocab | English tokens | Urdu tokens | Inflation | |---|---|---|---|---| | GPT-2 | ~50k | 320 | 1,999 | 5.39× | | Mistral v0.3 | ~32k | 359 | 1,554 | 3.73× | | Qwen2.5 | ~151k | 318 | 1,002 | 2.72× | Per character it’s starker: English gets ~5.35 characters per token on GPT-2, Urdu gets ~1.34. What surprised me is that vocabulary size isn’t the whole story — GPT-2 has a larger vocabulary than Mistral 50k vs 32k and does worse. What the tokenizer was trained on seems to matter at least as much. Two caveats I want to be upfront about: 30 sentences is a pilot, not a precise multiplier, and the corpus is formal register only — no conversational or Roman Urdu. Has anyone measured this for other South Asian languages? I’d like to extend the corpus to Pashto, Sindhi and Shahmukhi Punjabi.