{"slug": "language-models-and-languages-why-multilingual-is-an-illusion-of-equality", "title": "Language Models and Languages: Why \"Multilingual\" Is an Illusion of Equality", "summary": "A technical breakdown of how tokenizer training data skewed toward English — with English and code making up 95% of Llama 3's training data — creates measurable cost, context-window, and accuracy penalties for other languages. One study across 25 European languages and ten models found tokenization fertility ranges from about 1.2 tokens per word for English to roughly 3.1 for Greek and Maltese, and research on the HRM8K benchmark attributes the performance gap primarily to difficulty comprehending non-English input rather than weaker reasoning.", "body_md": "When people see ChatGPT or Claude answer fluently in Swahili or Thai, the natural impression is that the model \"knows\" all languages equally well. It doesn't — and the gap isn't cosmetic. It runs straight through the architecture of how these models read and process text. What follows is a breakdown of where that gap actually comes from, how large it is, and what's being done about it.\n\nAn LLM never reads text directly. Text is first cut into pieces (tokens) by a tokenizer, and only those tokens are what the model actually processes. Tokenizers are trained on corpora where English historically dominates — in Llama 3's training data, for instance, English and code made up 95% of the dataset, while every other language combined accounted for just 5%.\n\nBecause of this skew, the tokenizer ends up \"efficient\" for English and noticeably less efficient for everything else: models generate text one token at a time, and for languages poorly represented in the tokenizer's vocabulary, this makes generation slower and more computationally expensive.\n\nHow big is the gap in practice? One study covering 25 European languages and ten models put a concrete number on it: tokenization \"fertility\" ranges from roughly 1.2 tokens per word for English to roughly 3.1 for Greek and Maltese — a penalty of about 2.5x. Notably, the ranking of \"efficient\" vs. \"inefficient\" languages barely changes regardless of register — formal or casual text — with a correlation above 0.97 across styles.\n\nA similar picture shows up for languages using a different script. A comparison of several tokenizers on equivalent Polish and English text found that where Polish required 547 tokens, English needed only 377 on the same model (Mistral Small 3.2) — and this gap repeats across most tested models.\n\nTokenization inefficiency isn't only \"responses in non-English are slower.\" Three effects stack on top of each other:\n\n**Cost.** Since API pricing is billed per token, not per character, the same amount of meaning costs literally more in a \"token-expensive\" language. One breakdown describes a real case: a team running multilingual support saw its API bill triple in two weeks with no change to the prompt — the Brazilian and Indonesian user share simply grew, and their requests tokenized at 1.6x the per-character cost of English under the older vocabulary. Switching tokenizers cut the per-character token cost for those languages by 35% — without touching the model or the prompt.\n\n**Context window.** If the same meaning takes several times more tokens in another language, a model with a fixed context limit (say, 128k tokens) effectively fits much less real text for that language — less of a document, less conversation history.\n\n**Answer quality.** This isn't just a \"more expensive\" effect — it's also a \"worse\" effect. There's direct evidence of a correlation: the higher a language's tokenization compression ratio, the lower the model's task accuracy in that language. In other words, inefficient tokenization isn't a neutral side effect — it correlates with a real drop in quality.\n\nHere's where it gets more interesting than a simple \"more tokens, therefore worse.\"\n\nOne key study (the HRM8K benchmark, Korean and English) tested the hypothesis head-on: maybe it's not tokenization, maybe the model simply \"reasons\" worse in non-English? The answer turned out to be more specific: the performance gap stems primarily from difficulty comprehending non-English input, not from a limitation in the reasoning capability itself. The authors also directly challenged an earlier recommendation to force the model to reason in English regardless of the question's language — it turns out models are heavily influenced by the language of the input itself, not just the language of the reasoning chain.\n\nA similar picture shows up in domain-specific benchmarks. On an Irish-language benchmark built from real exam questions, the best-performing model scored 76.2% accuracy in English versus 55.8% in Irish, and produced a valid Irish response (one that didn't drift into English or break the language) less than 80% of the time.\n\nOne nuance that's often missed: the performance gap isn't uniform across task types. Evidence suggests the gap is especially pronounced in reasoning tasks dominated by high-resource languages (English foremost), but this pattern doesn't hold for factual questions or broader real-world user queries. A model might know the capital of Thailand perfectly well in Thai, but noticeably struggle with a multi-step logical problem in the same language.\n\nWorth approaching this from another angle — not through output benchmarks, but through what happens inside the model during generation. Anthropic has direct data on this, obtained through interpretability methods (an \"AI microscope\" — a way of looking directly at a model's internal activations).\n\nThe key finding isn't about English as an \"internal language of thought,\" but something more abstract: when analyzing how the model processes opposite-meaning words across languages, researchers found that the same internal features for \"smallness\" and \"oppositeness\" activate regardless of which language the input text is in. In other words, the model develops a shared, language-independent conceptual space, and any specific language is more of a \"surface encoding\" layered on top of that shared representation.\n\nThis doesn't mean all languages are represented equally well in that space — the shared space exists, but the quality of \"access\" to it through a given language still depends on how richly that language was represented during training. It's important not to conflate the two: the existence of a shared conceptual space explains why the model can transfer knowledge across languages at all, but it doesn't erase the fact that input in a less-represented language still passes through a noisier, less precise \"channel\" into that space.\n\nA separate, fairly unexpected finding from the same Anthropic research concerns not capability but behavior: they found that the values the model expresses shift systematically depending on the language of the conversation — for example, in English the model leans toward caution, rigor, depth, and candor, while in Arabic it leans toward deference, warmth, brevity, and task execution. This isn't about intelligence — it's that the model's \"character\" isn't language-neutral at all. Training data in different languages carries different cultural and stylistic norms, and the model absorbs them differently for each language.\n\nDeveloper communities occasionally circulate advice like \"write code-generation prompts in Chinese — it saves up to 40% in tokens.\" A recent study tested this directly on a real engineering benchmark (SWE-bench Lite) across several models and got an unambiguous result: the claimed Chinese token efficiency advantage did not hold up.\n\nThis is a useful lesson on its own: the intuition that a \"more compact\" script (logographic characters versus Latin letters) should automatically be more efficient for a model doesn't hold. A tokenizer isn't just a character counter — it's a separately trained system with its own quirks, and you can't predict its behavior by eye without actually measuring it.\n\nTwo directions stand out as attempts to solve this at the architecture level rather than patch it after the fact.\n\n**Dropping tokenization altogether.** Meta's Byte Latent Transformer doesn't tokenize input into discrete vocabulary IDs at all — instead it dynamically groups bytes into variable-length patches during training, with patch boundaries set by an entropy model. At comparable compute, an 8B-parameter BLT model matched a BPE-based Llama 3 8B baseline on standard benchmarks, while handling typos, code, and low-resource languages noticeably better. The point is that if a model has no privileged token vocabulary built on an English corpus in the first place, the structural inequality between languages disappears at the architecture level.\n\n**Targeted post-training instead of architectural change.** A more practical — and far cheaper — path is not rebuilding the model from scratch, but fine-tuning it specifically to close this gap. One approach (UST — Understand, Solve, Translate) uses English as an anchor language for reasoning and solution generation, and on a synthetic dataset of 130,000 examples achieved a 10.91% improvement on the HRM8K benchmark, cutting the cross-language gap from 11.6% down to 0.7%. Industry analysis backs this up: tokenizer inefficiency and English-centric reasoning account for 70-80% of model failures in non-English languages, and targeted post-training looks like the most cost-effective fix compared to a full architectural overhaul or scaling up model size.\n\nIf you're building a product for a multilingual audience — whether that's RealFeedApp serving news in multiple languages, or any other LLM-backed service — the language gap needs to be designed into the architecture up front, not filed under \"minor localization detail\":\n\nThe tokenizer is the least visible part of an LLM application's stack, which is exactly why it's the easiest thing to inherit unexamined along with your model choice. In practice, it's arguably the single most consequential hidden factor in how well your product performs outside of English.\n\n**Sources and further reading:**", "url": "https://wpnews.pro/news/language-models-and-languages-why-multilingual-is-an-illusion-of-equality", "canonical_source": "https://dev.to/den0011/language-models-and-languages-why-multilingual-is-an-illusion-of-equality-59ii", "published_at": "2026-09-10 15:24:46+00:00", "updated_at": "2026-09-10 15:45:28.493537+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "ai-research", "machine-learning"], "entities": ["Llama 3", "Mistral Small 3.2", "ChatGPT", "Claude", "HRM8K"], "alternates": {"html": "https://wpnews.pro/news/language-models-and-languages-why-multilingual-is-an-illusion-of-equality", "markdown": "https://wpnews.pro/news/language-models-and-languages-why-multilingual-is-an-illusion-of-equality.md", "text": "https://wpnews.pro/news/language-models-and-languages-why-multilingual-is-an-illusion-of-equality.txt", "jsonld": "https://wpnews.pro/news/language-models-and-languages-why-multilingual-is-an-illusion-of-equality.jsonld"}}