# Tokenizer fairness for Urdu: 5.39× more tokens on GPT-2, 2.72× on Qwen2.5 > Source: > Published: 2026-11-04 21:42:47+00:00 Hi all — I measured how many tokens Urdu costs compared with English for the same meaning, and wanted to share the numbers and ask for feedback. I built a small parallel corpus (30 sentence pairs, 17 domains, CC-BY-4.0) and tokenized both sides with three tokenizers: | Tokenizer | Vocab | English tokens | Urdu tokens | Inflation | |---|---|---|---|---| | GPT-2 | ~50k | 320 | 1,999 | **5.39×** | | Mistral v0.3 | ~32k | 359 | 1,554 | **3.73×** | | Qwen2.5 | ~151k | 318 | 1,002 | **2.72×** | Per character it’s starker: English gets ~5.35 characters per token on GPT-2, Urdu gets ~1.34. What surprised me is that vocabulary size isn’t the whole story — GPT-2 has a *larger* vocabulary than Mistral (50k vs 32k) and does worse. What the tokenizer was trained on seems to matter at least as much. Two caveats I want to be upfront about: 30 sentences is a pilot, not a precise multiplier, and the corpus is formal register only — no conversational or Roman Urdu. Has anyone measured this for other South Asian languages? I’d like to extend the corpus to Pashto, Sindhi and Shahmukhi Punjabi.