{"slug": "the-indic-token-tax-in-2026-measuring-tokenizer-disparity-and-its-cost-for-the", "title": "The Indic Token Tax in 2026: Measuring Tokenizer. Disparity and Its Cost for the Languages of India", "summary": "A study of nine public tokenizers, including those of GPT-4, GPT-4o, Qwen2.5, DeepSeek-V3, Mistral-Nemo and four Sarvam models, found that GPT-4o's tokenizer requires between 1.57 times (Hindi) and 5.01 times (Odia) the tokens of English for the same content across the 25 languages of India in FLORES-200, and 13.7 times for Santali in Ol Chiki script, while GPT-4's tokenizer needs up to 12.8 times. Sarvam-30B/105B was the only third-party tokenizer staying below 2.6 times on every language (median 1.53). Using official list prices accessed on 5 November 2026, the authors estimate a support bot handling 100,000 chats a month on GPT-4o would cost about INR 56,640 in English versus INR 102,829 in Marathi, and report that no tokenizer places more than 35% of its word-internal boundaries on morpheme boundaries (boundary F1 at most 0.48).", "body_md": "arXiv:2611.13058v1 Announce Type: new \nAbstract: Commercial language model APIs bill per token, so a tokenizer that needs more tokens for the same content in one language makes that language more expensive, slower, and able to fit less text in a fixed context window. We measure this disparity for nine public tokenizers, including those of GPT-4, GPT-4o, Qwen2.5, DeepSeek-V3, Mistral-Nemo and four Sarvam models, on all 25 languages of India available in FLORES-200 (2,009 professionally translated sentences aligned with English) and on 13,732 human-translated Hinglish-English social media pairs from PHINC. On GPT-4o's tokenizer the same content needs between 1.57 times (Hindi) and 5.01 times (Odia) the tokens of English, and 13.7 times for Santali in Ol Chiki script; GPT-4's tokenizer needs up to 12.8 times. Sarvam-30B/105B is the only third-party tokenizer that stays below 2.6 times on every language (median 1.53). In an illustrative scenario using official list prices accessed on 5 November 2026 and stated assumptions, a support bot handling 100,000 chats a month on GPT-4o would cost about INR 56,640 in English and INR 102,829 in Marathi; these are estimates, not measured production costs. Against an openly licensed Hindi and Marathi morphological segmentation set, no tokenizer places more than 35% of its word-internal boundaries on morpheme boundaries (boundary F1 at most 0.48). In a companion paper we present a tokenizer that reduces this tax. Code, pinned artifacts and raw token counts are available at https://github.com/HavonAI/indic-token-tax.", "url": "https://wpnews.pro/news/the-indic-token-tax-in-2026-measuring-tokenizer-disparity-and-its-cost-for-the", "canonical_source": "https://arxiv.org/abs/2611.13058", "published_at": "2026-11-11 05:00:00+00:00", "updated_at": "2026-11-11 05:19:36.824236+00:00", "lang": "en", "topics": ["natural-language-processing", "large-language-models", "ai-research", "artificial-intelligence"], "entities": ["GPT-4", "GPT-4o", "Qwen2.5", "DeepSeek-V3", "Mistral-Nemo", "Sarvam", "FLORES-200", "PHINC"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-indic-token-tax-in-2026-measuring-tokenizer-disparity-and-its-cost-for-the", "markdown": "https://wpnews.pro/news/the-indic-token-tax-in-2026-measuring-tokenizer-disparity-and-its-cost-for-the.md", "text": "https://wpnews.pro/news/the-indic-token-tax-in-2026-measuring-tokenizer-disparity-and-its-cost-for-the.txt", "jsonld": "https://wpnews.pro/news/the-indic-token-tax-in-2026-measuring-tokenizer-disparity-and-its-cost-for-the.jsonld"}}