cd /news/large-language-models/inherited-weights-measuring-the-toke… · home › topics › large-language-models › article
[ARTICLE · art-167425] src=arxiv.org ↗ pub= topic=large-language-models verified=true sentiment=· neutral

Inherited Weights: Measuring the Tokenization Premium and the Sovereignty Cost of Language Models for the Global South

A new arXiv paper (2611.13441v1) measures a "tokenization premium" across 31 languages and 7 tokenizers released in 2023-2024, finding premiums reach 10x for Amharic in older tokenizers and remain at 6.8x in the newest commercial one, while Indigenous South American languages pay between 1.6x and 2.7x under global tokenizers. The authors report that a vocabulary built from Portuguese lowers the premium for 11 of 13 Latin-script Global South languages relative to every global tokenizer tested, by about 24% for Guarani and Quechua, and they propose an eight-test black-box sovereignty audit a buyer can run without access to weights. The paper argues that accuracy per joule, calibration and selective risk should replace raw accuracy as design objectives under energy and liability constraints, and includes preliminary internal measurements from a proprietary architecture plus its data and measurement code.

by read1 min views1 publishedNov 11, 2026

arXiv:2611.13441v1 Announce Type: new Abstract: Most language models deployed outside the United States and China are fine-tuned descendants of a small number of pretrained families. We argue that fine-tuning leaves the decisive properties of a model inherited from its base: vocabulary, knowledge, cultural alignment, memorized data and license. We contribute (i) an updated, reproducible measurement of the tokenization premium across 31 languages and 7 tokenizers released in 2023-2024, on 27 articles of the Universal Declaration of Human Rights, with explicit correction for unknown-token collapse. Premiums reach 10x for Amharic in older tokenizers and remain at 6.8x in the newest commercial one; Indigenous South American languages pay between 1.6x and 2.7x under global tokenizers. A vocabulary built from Portuguese lowers the premium for 11 of 13 Latin-script Global South languages relative to every global tokenizer tested, by about 24% for Guarani and Quechua. (ii) An eight-test black-box sovereignty audit that a buyer can run without access to weights. (iii) A regime argument: under energy and liability constraints, accuracy per joule, calibration and selective risk should replace raw accuracy as design objectives, with metrics that abstention cannot inflate. We close with preliminary internal measurements from a proprietary architecture built around these objectives, reported with their limitations. Data and measurement code are included.

── more in #large-language-models 4 stories · sorted by recency
── more on @arxiv 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/inherited-weights-me…] indexed:0 read:1min 2026-11-11 · —