{"slug": "tokenizer-fairness-for-urdu-5-39x-more-tokens-on-gpt-2-2-72x-on-qwen2-5", "title": "Tokenizer fairness for Urdu: 5.39× more tokens on GPT-2, 2.72× on Qwen2.5", "summary": "A pilot study of 30 parallel English–Urdu sentence pairs across 17 domains found that Urdu text consumes 5.39× more tokens than English on GPT-2, 3.73× more on Mistral v0.3, and 2.72× more on Qwen2.5. The author reports that vocabulary size alone does not explain the gap: GPT-2's ~50k vocabulary is larger than Mistral's ~32k, yet GPT-2 performs worse, suggesting training data matters at least as much. Per character, English averages ~5.35 characters per token on GPT-2 versus ~1.34 for Urdu; the author cautions that 30 sentences is a pilot rather than a precise multiplier and covers only formal register, and plans to extend the corpus to Pashto, Sindhi and Shahmukhi Punjabi.", "body_md": "Hi all — I measured how many tokens Urdu costs compared with English for the same meaning, and wanted to share the numbers and ask for feedback.\n\nI built a small parallel corpus (30 sentence pairs, 17 domains, CC-BY-4.0) and tokenized both sides with three tokenizers:\n\n| Tokenizer | Vocab | English tokens | Urdu tokens | Inflation | \n|---|---|---|---|---|\n| GPT-2 | ~50k | 320 | 1,999 | **5.39×** | \n| Mistral v0.3 | ~32k | 359 | 1,554 | **3.73×** | \n| Qwen2.5 | ~151k | 318 | 1,002 | **2.72×** | \n\nPer character it’s starker: English gets ~5.35 characters per token on GPT-2, Urdu gets ~1.34.\n\nWhat surprised me is that vocabulary size isn’t the whole story — GPT-2 has a *larger* vocabulary than Mistral (50k vs 32k) and does worse. What the tokenizer was trained on seems to matter at least as much.\n\nTwo caveats I want to be upfront about: 30 sentences is a pilot, not a precise multiplier, and the corpus is formal register only — no conversational or Roman Urdu.\n\nHas anyone measured this for other South Asian languages? I’d like to extend the corpus to Pashto, Sindhi and Shahmukhi Punjabi.", "url": "https://wpnews.pro/news/tokenizer-fairness-for-urdu-5-39x-more-tokens-on-gpt-2-2-72x-on-qwen2-5", "canonical_source": "https://discuss.huggingface.co/t/tokenizer-fairness-for-urdu-5-39x-more-tokens-on-gpt-2-2-72x-on-qwen2-5/191440#post_1", "published_at": "2026-11-04 21:42:47+00:00", "updated_at": "2026-11-04 21:51:50.112287+00:00", "lang": "en", "topics": ["natural-language-processing", "large-language-models", "ai-research"], "entities": ["GPT-2", "Mistral v0.3", "Qwen2.5", "Urdu", "Pashto", "Sindhi", "Shahmukhi Punjabi"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/tokenizer-fairness-for-urdu-5-39x-more-tokens-on-gpt-2-2-72x-on-qwen2-5", "markdown": "https://wpnews.pro/news/tokenizer-fairness-for-urdu-5-39x-more-tokens-on-gpt-2-2-72x-on-qwen2-5.md", "text": "https://wpnews.pro/news/tokenizer-fairness-for-urdu-5-39x-more-tokens-on-gpt-2-2-72x-on-qwen2-5.txt", "jsonld": "https://wpnews.pro/news/tokenizer-fairness-for-urdu-5-39x-more-tokens-on-gpt-2-2-72x-on-qwen2-5.jsonld"}}