{"slug": "what-an-urdu-sentence-costs-in-tokens", "title": "What an Urdu sentence costs in tokens", "summary": "A developer built an original 30-pair English–Urdu parallel corpus and benchmarked tokenizer fertility across three models, finding the same meanings cost 320 tokens in English versus 1,999 in Urdu on GPT-2 (5.39× inflation), 3.73× on Mistral v0.3, and 2.72× on Qwen2.5. On GPT-2, English packs 5.35 characters per token while Urdu gets 1.34, meaning a 128k context holds roughly a fifth as much Urdu as English. The corpus is released CC-BY-4.0 on Hugging Face as urdu-english-parallel.", "body_md": "Here are two sentences that mean exactly the same thing:\n\nThe government announced a new policy on electricity prices yesterday.\n\nحکومت نے کل بجلی کی قیمتوں سے متعلق نئی پالیسی کا اعلان کیا۔\n\nTen words in English, thirteen in Urdu. If you're paying for an LLM, you'd expect them to cost roughly the same.\n\nMeasured across 30 pairs like this one, the same meanings cost **320 tokens in English and 1,999 in Urdu** on GPT-2.\n\nThat's not a rounding error. That's the price of writing in a different script.\n\nEvery LLM interaction is measured in tokens — you're billed in them, and your context window is counted in them. Words and characters are irrelevant to the meter.\n\nTokenizers are trained on whatever text their creators had, and that text is overwhelmingly English. When a tokenizer meets a script it barely saw during training, it falls back to breaking text into small pieces — sometimes into raw bytes.\n\nThe result is that \"one page\" of Urdu is not one page of tokens. It's five pages' worth.\n\nI wanted a real number instead of a vibe, so I built a small benchmark to measure it.\n\nThe measurement is called **fertility**: tokens divided by words. Compare the same content in two languages, and the ratio tells you how much more you pay.\n\nThe key requirement is that the two texts must mean the same thing — otherwise you're measuring content, not encoding. So I wrote a parallel corpus: 30 sentences, each written in both English and Urdu, spanning 17 everyday domains (news, medical, legal, technical, agriculture, and so on).\n\nI couldn't use FLORES+, the standard benchmark for this work. Its access gate explicitly forbids re-hosting it anywhere web crawlers can reach, and a public Space or dataset would violate that. So the corpus is original, released CC-BY-4.0, and freely re-hostable: [urdu-english-parallel](https://huggingface.co/datasets/abdulrauf5121/urdu-english-parallel).\n\nCorpus shape: 30 pairs, 283 English words / 1,711 characters vs 328 Urdu words / 2,680 characters. All strings verified NFC, with no zero-width joiners and no Arabic presentation forms — those inflate token counts for reasons that have nothing to do with vocabulary, and they are easy to overlook.\n\nThree tokenizers, spanning a wide range of vocabulary sizes:\n\n| Tokenizer | Vocab | English tokens | Urdu tokens | Urdu inflation | \n|---|---|---|---|---|\n| GPT-2 | ~50k | 320 | 1,999 | **5.39×** | \n| Mistral v0.3 | ~32k | 359 | 1,554 | **3.73×** | \n| Qwen2.5 | ~151k | 318 | 1,002 | **2.72×** | \n\nSame 30 sentences, same meanings, in every row.\n\nInflation is Urdu fertility ÷ English fertility. A value of 5.39× means the same meaning costs you more than five times as many tokens.\n\nFertility is abstract. Here it is per character:\n\n|  | Characters per token | \n|---|---|\n| English (GPT-2) | **5.35** | \n| Urdu (GPT-2) | **1.34** | \n\nEnglish gets more than five characters packed into every token. Urdu gets **one and a third**.\n\nThat is close to character-level: on this tokenizer, Urdu is being encoded nearly one letter at a time rather than one word at a time. A thousand-character Urdu page costs about **746 tokens** on GPT-2; the same number of characters in English costs about **187**.\n\nA 128,000-token context sounds enormous. Here's what it holds:\n\n| Tokenizer | Words of English | Words of Urdu | \n|---|---|---|\n| GPT-2 | ~113,000 | ~21,000 | \n| Qwen2.5 | ~114,000 | ~41,900 | \n\n(English capacity differs slightly per row because English fertility differs slightly — 1.13 vs 1.12.)\n\nOn the English-trained tokenizer, a \"128k\" model holds **roughly a fifth** of the Urdu it holds of English. Put a 200-page Urdu document in front of it and you're truncating most of it. In a RAG pipeline, the same chunk costs five times as much, so you retrieve far less for the same budget.\n\nAnd the billing follows the context window: at any per-token price, Urdu documents cost 2.7–5.4× more to process than English ones of the same substance.\n\nI expected this to be a pure vocabulary-size story. It isn't.\n\nGPT-2 has **50k** vocabulary and 5.39× inflation. Mistral has only **32k** and still does better at 3.73×. Qwen2.5 with 151k is best at 2.72×.\n\nSo vocabulary size helps, but *what the tokenizer was trained on* appears to matter at least as much. My reading is that Mistral's tokenizer saw far more non-English text than GPT-2's, and that diversity outweighs an 18k-vocabulary deficit. Either way the conclusion holds: you cannot fix script fairness by scaling the vocabulary alone.\n\nIf you're building for Urdu users today:\n\nEverything runs in your browser — no server, no API key, your text never leaves the page:\n\nThe Space shows the actual subword pieces. Some are pink: those are single Urdu characters split across two tokens. That visual is the whole argument in one glance.\n\nThis isn't only about Urdu. Every language outside the tokenizer's training distribution pays a tax — in money, in context, in latency, in what fits in a prompt. The tax is invisible unless you measure it, and it compounds for exactly the users who already have less compute directed at their language.\n\nIf you speak a language that isn't well represented in tokenizers, the most useful thing you can do is measure it. The corpus format is 30 sentences and a JSON file. Add your language, run the Space, publish the number.\n\nThe tax is real. It should at least be visible.", "url": "https://wpnews.pro/news/what-an-urdu-sentence-costs-in-tokens", "canonical_source": "https://dev.to/abdul_raufghulamnabi_65/what-an-urdu-sentence-costs-in-tokens-1mk6", "published_at": "2026-11-04 21:16:46+00:00", "updated_at": "2026-11-04 21:22:50.345299+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "ai-research", "ai-infrastructure"], "entities": ["GPT-2", "Mistral", "Qwen2.5", "Hugging Face", "FLORES+"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/what-an-urdu-sentence-costs-in-tokens", "markdown": "https://wpnews.pro/news/what-an-urdu-sentence-costs-in-tokens.md", "text": "https://wpnews.pro/news/what-an-urdu-sentence-costs-in-tokens.txt", "jsonld": "https://wpnews.pro/news/what-an-urdu-sentence-costs-in-tokens.jsonld"}}