Hi all — I measured how many tokens Urdu costs compared with English for the same meaning, and wanted to share the numbers and ask for feedback.
I built a small parallel corpus (30 sentence pairs, 17 domains, CC-BY-4.0) and tokenized both sides with three tokenizers:
| Tokenizer | Vocab | English tokens | Urdu tokens | Inflation |
|---|---|---|---|---|
| GPT-2 | ~50k | 320 | 1,999 | 5.39× |
| Mistral v0.3 | ~32k | 359 | 1,554 | 3.73× |
| Qwen2.5 | ~151k | 318 | 1,002 | 2.72× |
Per character it’s starker: English gets ~5.35 characters per token on GPT-2, Urdu gets ~1.34.
What surprised me is that vocabulary size isn’t the whole story — GPT-2 has a larger vocabulary than Mistral (50k vs 32k) and does worse. What the tokenizer was trained on seems to matter at least as much.
Two caveats I want to be upfront about: 30 sentences is a pilot, not a precise multiplier, and the corpus is formal register only — no conversational or Roman Urdu.
Has anyone measured this for other South Asian languages? I’d like to extend the corpus to Pashto, Sindhi and Shahmukhi Punjabi.