# Logical word count: a cheap, tokenizer-free measure of text length that tracks LLM token counts across languages, code, and mixed content > Source: > Published: 2026-07-12 22:47:23+00:00 **Logical word count** is a cheap, deterministic, tokenizer-free measure of text length that stays roughly proportional to LLM token counts across natural languages, code, and mixed content — while remaining equal to ordinary word count for plain English prose. It is defined in two rules: **Wide (East Asian) characters count as ½ logical word each.** This covers Chinese characters, Japanese kana, Korean Hangul, and CJK punctuation.**Everything else is split on whitespace, and the word count is clamped so the average word length stays between 3 and 6 characters.**Text with very long “words” (compounds, identifiers, URLs, minified code) is counted as`chars / 6` ; text with very short words is counted as`chars / 3` ; typical prose passes through unchanged. A reference implementation is ~15 lines with no dependencies (see [Reference implementation](https://gist.github.com/starred.atom#reference-implementation)). Empirically, tokens per logical word stays within a ~2× band for every tokenizer tested — 1.36–2.24 for the GPT-5.6 / o200k family — across ten languages plus source code and machine formats, versus a 24× spread for literal word count and a 3× spread for the common `chars ÷ 4` rule on the same corpus (see [Validation](https://gist.github.com/starred.atom#validation)). Word count is the standard unit for measuring text length, but it breaks in two distinct ways: it can under-count content volume, and it can over-count it. CJK text has few or no word-delimiting spaces; Unicode’s own segmentation standard notes that word boundaries in Chinese, Japanese, or Thai require dictionary lookup. 1 A 500-character Chinese paragraph counts as a handful of whitespace-delimited “words” despite carrying the information of roughly 250 English words. In our test corpus, an 8,000-character Chinese Wikipedia extract contains just 165 whitespace-delimited tokens. German compounds like *Donaudampfschifffahrtsgesellschaft* 2 pack five lexical units into one “word”; agglutinative languages like Finnish average 8+ characters per whitespace-delimited word versus ~5.5 for English. Identical content, materially different word counts. Code is the hardest case, for two reasons: **Symbols are rarely space-separated.** The expression `a+b*clamp(x,lo,hi)` is one whitespace-delimited “word” but contains eight semantic tokens. Minified JavaScript averages 44 characters per whitespace-delimited “word” in our corpus; CSV rows and JSON average 10–14. **Identifiers vary wildly in length.** `x` and `calculateAverageResponseTime` are both one word, but the latter contains five semantic units. Code with long descriptive names is under-counted relative to its actual volume. Documents that combine prose, code blocks, tables, links, and CJK text have no single word-counting strategy that works across all sections. A README with badges, URLs, and code samples measures 11.7 characters per “word” — double typical prose. Token counts solve the normalization problem by construction, but they are not stable across model providers, they require a tokenizer vocabulary, and they are orders of magnitude more expensive to compute than a character scan. **Token counts vary across providers — and across model generations within a provider.** The same 8,000-character samples measured against the latest production tokenizers (July 2026):[3](https://gist.github.com/starred.atom#user-content-fn-token-variance-310cd646f058931c1f7e1f9c14b67030) | Sample | GPT-5.6 | Claude Opus 4.8 / Fable 5 | DeepSeek V4 | max deviation | |---|---|---|---|---| | English Wikipedia | 1,657 | 2,471 | 1,664 | 49% | | German Wikipedia | 1,816 | 3,695 | 2,068 | 103% | | Chinese Wikipedia | 5,351 | 7,348 | 4,333 | 70% | | Japanese Wikipedia | 5,097 | 6,403 | 4,343 | 47% | | Korean Wikipedia | 4,529 | 7,916 | 5,026 | 75% | | Thai Wikipedia | 3,520 | 7,107 | 3,238 | 119% | | Python code | 1,853 | 2,810 | 1,893 | 52% | | TypeScript code | 1,907 | 3,518 | 2,004 | 84% | Even for English the tokenizers diverge by 49%, and for German by 103%. Nor are counts stable over time: Anthropic’s newest tokenizer (shared by Claude Opus 4.8, Fable 5, and Sonnet 5) spends ~37% more tokens on the same English text than the Opus 4.6 tokenizer did, and ~2× on Thai. Academic work documents far larger gaps across languages: the same content can take up to 15× more tokens in some languages than in English. 4 A “token count” is therefore not a property of a text — it is a property of a (text, model) pair. Any model-independent measure has this cross-tokenizer variance as a floor on its achievable accuracy, and that is fine: the goal is a stable measure of *content volume*that tokens track within a predictable band, not a replacement for a tokenizer when exact counts matter. **Token counts also require a vocabulary.** Computing them means shipping or calling the target model’s tokenizer — a dependency that may not exist where length is measured (a database check constraint, a CLI, an editor plugin, a non-Python runtime) and that changes when models change. Given a text string: **Wide characters.** Count the characters whose Unicode East Asian Width is Wide or Fullwidth— Han ideographs, Hiragana, Katakana, Hangul syllables, CJK punctuation, and fullwidth forms. Call this[5](https://gist.github.com/starred.atom#user-content-fn-uax11-310cd646f058931c1f7e1f9c14b67030)`wide` . Each contributes**½** logical word. For splitting purposes, treat each wide character as if it were surrounded by spaces.**Remaining text.** Split what remains on whitespace. Let`words` be the number of resulting tokens and`chars` the total count of their characters (Unicode code points). Clamp the word count so that average word length stays in`[L, U] = [3, 6]` : ``` clamped = clamp(words, chars / 6, chars / 3) = max(chars / 6, min(chars / 3, words)) logical_words = round(wide / 2 + clamped) ``` Empty or whitespace-only text has 0 logical words. The clamp activates only when average characters per word falls outside `[3, 6]` . Most English and European prose runs 4.5–5.5 characters per whitespace-delimited word (punctuation attached), so for ordinary prose **logical word count equals literal word count** — the measure is backward-compatible where word count already works. Long-word text (German and Finnish compounds, identifiers, URLs, minified code, CSV rows) is counted as `chars / 6` ; degenerate short-word text (e.g. tables of single digits) is capped at `chars / 3` . In practice the upper bound does nearly all the work; the lower bound is a safety rail that real content rarely hits. **Canonical definition:** East_Asian_Width ∈ {W, F} (available in Python’s`unicodedata` , ICU, and most Unicode libraries). Where the property is unavailable, a range-based approximation — the Han and Hangul scripts plus the CJK punctuation-through-Katakana blocks (U+3000–30FF), Katakana phonetic extensions (U+31F0–31FF), and fullwidth forms (U+FF00–FFEF) — produces*identical*counts on all our CJK test samples (kana must be matched by block, not script property: ー U+30FC and ・ U+30FB are Script=Common but Wide). Rare code points (halfwidth Katakana, some symbols) may still differ between the two definitions; the effect on real text is well under 1%.**Korean:** Hangul is space-delimited, but each syllable block encodes 2–3 jamo and tokenizers price Korean like Chinese and Japanese (~1.4 chars/token). Weighting Hangul syllables at ½ overrides the space-based count and empirically aligns Korean with every other language (K = 1.57 vs. 2.54 without the rule; see[Validation](https://gist.github.com/starred.atom#validation)). The localization industry’s GMX-V standard likewise groups Korean with the logographic scripts.[6](https://gist.github.com/starred.atom#user-content-fn-gmxv-310cd646f058931c1f7e1f9c14b67030)**Emoji** are East_Asian_Width Wide, so they count ½ each. Tokenizers typically spend 1–3 tokens per emoji; at realistic emoji densities the error is negligible. Independent conventions, developed for different purposes, converge on roughly two CJK characters per English-word-equivalent: | Convention | Effective rate | Purpose | |---|---|---| | Translation-industry rule of thumb | ~1,000 Japanese chars ≈ 400–500 English words; 1,000 Chinese chars ≈ 650–750 English words | Quoting/billing | | Japanese 原稿用紙 (genkō yōshi) manuscript page | 400 chars ≈ 200–250 English words | Publishing | | Twitter/X character limit | CJK weighted 2× (280-unit limit = 140 CJK chars) | Length limiting | | Medium & Hugo reading speed | 500 CJK chars/min vs. ~212–265 words/min ≈ 2–2.4 chars per word | Reading time | | ICU dictionary segmentation | Chinese words average ~1.5–2 chars | Linguistic segmentation | | Modern LLM tokenizers (measured) | one CJK char ≈ 0.7 tokens ≈ half the ~1.35 tokens of an English word | Token accounting | The tokenizer alignment is the one this spec is calibrated to: weighting CJK characters at ½ word puts Chinese, Japanese, and Korean at 1.54–1.57 tokens per logical word under o200k — statistically indistinguishable from French (1.50) and German (1.57). GMX-V, the localization industry’s published volume standard, uses somewhat higher divisors for *source-word* counts (Chinese ÷ 2.8, Japanese ÷ 3.0, Korean ÷ 3.3) 6 — it estimates the number of words a native segmentation would find, whereas logical word count (like the billing ratios above) targets volume equivalence. Simple word-counting tools mostly land on 1× instead: Microsoft Word and memoQ count each CJK character as one word, and Unicode’s default word segmentation breaks between every pair of ideographs. [12](https://gist.github.com/starred.atom#user-content-fn-word-count-tools-310cd646f058931c1f7e1f9c14b67030)Half a word per character sits deliberately between “one word per character” (too high: overweights CJK ~2× in any cross-language comparison) and GMX-V’s ~⅓ (too low for token tracking). [1](https://gist.github.com/starred.atom#user-content-fn-uax29-310cd646f058931c1f7e1f9c14b67030)BPE tokenizers merge frequent byte sequences until each token carries roughly one “meaning-sized” chunk. 13 Measured on large samples, modern large-vocabulary tokenizers (o200k, cl100k, Llama 3, Gemma 2) all land near **4.2–4.3 characters per token for English prose** and 4.0–4.8 for source code; vendor rules of thumb agree (OpenAI: ~4 chars or ¾ of a word per token; Anthropic: ~3.5 chars; Google: ~4 chars). [14](https://gist.github.com/starred.atom#user-content-fn-chars-per-token-310cd646f058931c1f7e1f9c14b67030)(Anthropic’s newest tokenizer is denser — ~2.7 characters per token on our English sample — but uniformly so across content types, which shifts K by a constant factor without widening the band the clamp is designed to hold.) [15](https://gist.github.com/starred.atom#user-content-fn-vendor-rules-310cd646f058931c1f7e1f9c14b67030)The bounds bracket that range without a vocabulary table: **U = 6** is the point where long-word content (compounds, identifiers, URLs) stops gaining count from packing more characters into each word.`chars / 6` tracks the ~4–5 chars/token that tokenizers actually spend on such content, at a stable ratio of ~1.3–1.5 tokens per logical word.**L = 3** caps degenerate short-word content. Real prose almost never averages below 3 (even Korean, with dense syllable blocks, averages 3.5); the bound exists so that pathological inputs (single-character tables, heavily spaced tokens) cannot inflate the count more than ~1.5× over prose.- Everything between 3 and 6 chars/word passes through, preserving `logical_words = literal words` for ordinary prose in English, French, Spanish, Russian, Hebrew, and most other spaced languages. Tightening the bounds (e.g. `[3, 5]` ) narrows the token-tracking band another ~10% but sacrifices exact backward compatibility with English word count; loosening them (e.g. `[2, 7]` ) admits outliers without benefit. `[3, 6]` with weight ½ is the recommended convention; all three constants are parameters in the reference implementation for applications that want to recalibrate. Python (canonical): ``` python import unicodedata def logical_word_count(text: str, lower: float = 3.0, upper: float = 6.0, wide_weight: float = 0.5) -> int: wide = 0 rest = [] for c in text: if unicodedata.east_asian_width(c) in ("W", "F"): wide += 1 rest.append(" ") else: rest.append(c) words = "".join(rest).split() chars = sum(len(w) for w in words) clamped = max(chars / upper, min(chars / lower, len(words))) return round(wide * wide_weight + clamped) ``` TypeScript (script-range approximation of East_Asian_Width; identical results on real-world text): ``` js const WIDE = /[\p{Script=Han}\p{Script=Hangul}\u3000-\u30FF\u31F0-\u31FF\uFF00-\uFFEF]/u; function logicalWordCount( text: string, lower = 3, upper = 6, wideWeight = 0.5, ): number { let wide = 0; const rest: string[] = []; for (const ch of text) { // iterates code points, not UTF-16 units if (WIDE.test(ch)) { wide++; rest.push(" "); } else { rest.push(ch); } } const words = rest.join("").split(/\s+/).filter(Boolean); const chars = words.reduce((n, w) => n + [...w].length, 0); const clamped = Math.max(chars / upper, Math.min(chars / lower, words.length)); return Math.round(wide * wideWeight + clamped); } ``` Both are O(n), allocation-light, and dependency-free. Measured with the companion script (`validate_lwc.py` ) on 8,000-character samples: Wikipedia extracts in ten languages (Hebrew and Thai are reported under [Limitations](https://gist.github.com/starred.atom#limitations)), real source files, minified JS, CSV, JSON, and a mixed-content README. Token counts are GPT-5.6 via the OpenAI API (the o200k tokenizer family); K = tokens / logical words. | Sample | chars | literal words | logical words | tokens | K | |---|---|---|---|---|---| | English Wikipedia | 6,775 | 1,219 | 1,219 | 1,657 | 1.36 | | French Wikipedia | 6,756 | 1,222 | 1,222 | 1,838 | 1.50 | | German Wikipedia | 6,921 | 1,063 | 1,154 | 1,816 | 1.57 | | Russian Wikipedia | 6,998 | 995 | 1,166 | 1,939 | 1.66 | | Finnish Wikipedia | 7,104 | 868 | 1,184 | 2,309 | 1.95 | | Chinese Wikipedia | 7,807 | 165 | 3,400 | 5,351 | 1.57 | | Japanese Wikipedia | 7,642 | 300 | 3,300 | 5,097 | 1.54 | | Korean Wikipedia | 6,181 | 1,781 | 2,888 | 4,529 | 1.57 | | Python code | 5,709 | 862 | 952 | 1,853 | 1.95 | | TypeScript code | 6,800 | 865 | 1,133 | 1,907 | 1.68 | | Markdown README | 7,314 | 627 | 1,219 | 2,178 | 1.79 | | Minified JavaScript | 7,825 | 176 | 1,304 | 2,889 | 2.22 | | CSV data | 7,462 | 539 | 1,244 | 2,786 | 2.24 | | JSON | 7,037 | 560 | 1,173 | 2,168 | 1.85 | Prose, code, and CJK all land in **K = 1.36–1.95** (a 1.43× band). Punctuation-dense machine formats run hotter, up to ~2.2. Compare the alternatives on the identical corpus: | Measure | tokens ÷ measure, across samples | spread | |---|---|---| logical words | 1.36 – 2.24 | 1.65× | | literal words | 1.36 – 32.4 | 24× | | chars ÷ 4 | 0.98 – 2.9 | 3.0× | The same holds for the other tokenizer families. On the identical corpus (including the Hebrew and Thai limitation cases), tokens per logical word spans a 2.0× band for GPT-5.6, 2.1× for DeepSeek V4, and 2.9× for the Claude Opus 4.8 / Fable 5 tokenizer — each centered on its own K — while literal word count spans 19–24× on every one of them. The remaining variation is largely irreducible: the tokenizers themselves disagree by 37–119% on identical text (see the table in [Motivation](https://gist.github.com/starred.atom#why-not-just-use-token-counts)), so no model-free measure can track any single model much more tightly than models track each other. For a quick estimate against o200k-family tokenizers (GPT-5.x; DeepSeek V4 runs ~10% higher): ``` estimated_tokens ≈ logical_words × 1.6 ``` This lands within roughly ±25% across prose (any script), code, and markdown. Useful refinements: **Clean English prose:** use ×1.35 — which is the familiar “1 word ≈ 4⁄3 tokens” vendor rule,recovered exactly because logical words equal literal words for such text.[15](https://gist.github.com/starred.atom#user-content-fn-vendor-rules-310cd646f058931c1f7e1f9c14b67030)**Punctuation-dense machine formats**(minified code, CSV, logs): use ×2.2, or accept under-estimation.** Model-specific correction:**apply a per-tokenizer factor — DeepSeek V4 runs ~1.1× o200k, and Anthropic’s newest tokenizer (Claude Opus 4.8 / Fable 5 / Sonnet 5) runs ~1.5–2× o200k depending on content, i.e. tokens ≈ logical_words × 2.75 overall. This correction is roughly constant per model, unlike the correction needed for raw word count, which varies by an order of magnitude with text type. When exact counts matter (billing, hard context limits), use the model’s tokenizer; logical word count is for the much larger class of uses where a stable, cheap, model-free number is worth ±25%. **Cheap:** O(n) single pass, no vocabulary, no dependencies.**Deterministic and model-free:** same input, same count, forever.**Backward-compatible:** equals literal word count for ordinary English (and most European-language) prose.**Script-aware but language-blind:** uses only character properties; no language detection, dictionaries, or segmentation models.**Monotonic:** appending non-whitespace content never decreases the count.**Additive:** splitting text at whitespace boundaries and summing the parts’ counts approximates the whole (within rounding), so counts can be computed incrementally or in parallel.**Bounded token ratio:** every tokenizer tested spends a near-constant number of tokens per logical word — within a ~2–3× band across all content types, versus ~20–24× for literal word count. Stated plainly, since a measure is only trustworthy if its failure modes are known: **Unspaced non-CJK scripts**(Thai, Lao, Khmer, Myanmar) are not covered by the wide-character rule; they fall to the clamp’s`chars / 6` floor. That happens to match GMX-V’s convention for Thai (chars ÷ 6),but o200k still spends ~2.75 tokens per logical word on Thai (Anthropic’s newest tokenizer, ~5.6) — high fertility for these scripts is a property of the tokenizers, not of the measure. Applications centered on these scripts should add a per-script weight (Thai ≈ ¼–⅓ word/char) by the same method used here for CJK.[6](https://gist.github.com/starred.atom#user-content-fn-gmxv-310cd646f058931c1f7e1f9c14b67030)**Non-Latin alphabetic scripts**(Hebrew, Greek, Devanagari, …) count correctly as*volume*— their word counts are true word counts — but tokenizers charge them 1.9–2.3 tokens per logical word versus 1.35 for English (o200k; other tokenizer families show the same skew at their own scale). This is the well-documented tokenizer-fairness gap,[4](https://gist.github.com/starred.atom#user-content-fn-petrov-310cd646f058931c1f7e1f9c14b67030)and logical word count deliberately does not bend the volume measure to compensate for it.[16](https://gist.github.com/starred.atom#user-content-fn-ahia-310cd646f058931c1f7e1f9c14b67030)**Machine formats**(minified code, CSV, dense JSON) run ~1.4× hotter in tokens than prose. The clamp keeps them bounded (literal word count is off by 24× here), but a single multiplier will under-estimate their token cost.**It is not a linguistic word count.** For Chinese, ½ word per character approximates dictionary segmentation (Chinese words average 1.5–2 characters), but no claim is made that`logical_words` matches what a human or a segmenter would count as words in any specific language. It is a normalized volume unit, like a “standard page” — not a lexical analysis. No existing term or formula was found for this specific construction (a chars-per-word clamp plus a wide-character weight), and the name “logical word count” appears to be unclaimed. ("Standard word" already means 5 keystrokes in typing measurement; 17 “weighted word count” means fuzzy-match discounts in CAT tools.) The pieces, however, have deep roots: **GMX-V (LISA → ETSI GS LIS 004)**— the localization industry’s word/character count standard — is the closest precedent. Version 1.0 (2007) refused to segment unspaced scripts and required character counts; version 2.0 (2012) added normative per-script divisors to convert characters to words: Chinese ÷ 2.8, Japanese ÷ 3.0, Korean ÷ 3.3, Thai ÷ 6, described as “acknowledged best practice within the Localization Industry.”Logical word count generalizes the same move — chars ÷ factor — and extends it to code and mixed content via the clamp.[6](https://gist.github.com/starred.atom#user-content-fn-gmxv-310cd646f058931c1f7e1f9c14b67030)**CJK character-as-word conventions:** Microsoft Word’s “Words” statistic counts each East Asian character as a word (with a separate Asian-characters statistic); memoQ computes source words as Asian characters + non-Asian words; WordPress locale packs switch Japanese and Chinese to character counting outright.[12](https://gist.github.com/starred.atom#user-content-fn-word-count-tools-310cd646f058931c1f7e1f9c14b67030)**Weighted length limits:** Twitter/X’s 280-unit limit weights code points beyond a small Latin-ish set (roughly through U+10FF plus general punctuation) at 2, so CJK posts max out at 140 characters — a shipped, planet-scale precedent for weighting wide characters at exactly 2× Latin.[9](https://gist.github.com/starred.atom#user-content-fn-twitter-310cd646f058931c1f7e1f9c14b67030)**Reading-time estimators:** Medium uses ~265 words/min for spaced languages and 500 characters/min for CJK; Hugo and the`reading-time` npm package (used by Gatsby, Ghost analogues) count CJK per-character against a ~2.4× higher rate.[10](https://gist.github.com/starred.atom#user-content-fn-medium-310cd646f058931c1f7e1f9c14b67030)**Character-based billing units:** Germany’s court-interpreting law (§ 11 JVEG) prices translation per 55 keystrokes ("je angefangene 55 Anschläge"); publishing standard pages (Normseite: 1,500–1,800 chars) and typing’s 5-character “standard word” all define length by characters when word counts are unreliable or unfair.[18](https://gist.github.com/starred.atom#user-content-fn-jveg-310cd646f058931c1f7e1f9c14b67030)Amazon’s KENPC normalizes e-book length to a synthetic page unit for paying authors.[17](https://gist.github.com/starred.atom#user-content-fn-wpm-310cd646f058931c1f7e1f9c14b67030)[19](https://gist.github.com/starred.atom#user-content-fn-kenpc-310cd646f058931c1f7e1f9c14b67030)**Character length as a text statistic:** the Automated Readability Index (1967) used chars/word as a machine-friendly complexity proxy precisely because characters were cheap to count.[20](https://gist.github.com/starred.atom#user-content-fn-ari-310cd646f058931c1f7e1f9c14b67030)**Tokenizer research:** subword*fertility*(tokens per word) is the standard metric for tokenizer efficiency;the unfairness of token counts across languages is documented up to 15×.[21](https://gist.github.com/starred.atom#user-content-fn-fertility-310cd646f058931c1f7e1f9c14b67030)[4](https://gist.github.com/starred.atom#user-content-fn-petrov-310cd646f058931c1f7e1f9c14b67030)BPE itself[16](https://gist.github.com/starred.atom#user-content-fn-ahia-310cd646f058931c1f7e1f9c14b67030)solves normalization by construction, at the cost of a vocabulary and model dependence.[13](https://gist.github.com/starred.atom#user-content-fn-sennrich2016-310cd646f058931c1f7e1f9c14b67030)**Tokenizer-free token estimators**(the`tokenx` npm package; LangChain’s`count_tokens_approximately` ; Semantic Kernel’s chars ÷ 4) estimate tokens for a*specific*model family from character ratios.Logical word count differs in intent: it is a stable text measure first — human-legible, word-count compatible — that happens to track tokens within a known band, rather than a model-specific predictor.[22](https://gist.github.com/starred.atom#user-content-fn-estimators-310cd646f058931c1f7e1f9c14b67030) All measurements in this document come from the companion script `validate_lwc.py` (published alongside this document), which fetches the public samples, computes literal words, logical words, and exact token counts via provider APIs, and prints the comparison tables: ``` uv run validate_lwc.py # o200k locally; other providers with API keys uv run validate_lwc.py --lower 3 --upper 6 --wide-weight 0.5 uv run validate_lwc.py --list-samples ``` With no API keys, OpenAI counts fall back to local tiktoken (o200k_base — measured identical to the API counts modulo ~6 tokens of message framing). Set `OPENAI_API_KEY` , `ANTHROPIC_API_KEY` , and/or `DEEPSEEK_API_KEY` to reproduce the cross-model table with exact provider counts; Gemini and xAI models are supported via `--models` with their keys. The script’s dependencies are pinned under a two-week cooling-off policy (uv’s `exclude-newer` ). ## Footnotes - Unicode Standard Annex #29, “Unicode Text Segmentation,” §4 Word Boundaries: default rules break between every pair of ideographs (rule WB999), and “reliable detection of word boundaries in languages such as Thai, Lao, Chinese, or Japanese requires the use of dictionary lookup or other mechanisms.” [https://unicode.org/reports/tr29/](https://unicode.org/reports/tr29/)[↩](https://gist.github.com/starred.atom#user-content-fnref-uax29-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-uax29-2-310cd646f058931c1f7e1f9c14b67030)2 - Danube–steam–ship–journey–company; the Erste Donau-Dampfschiffahrts-Gesellschaft (founded 1829) is the canonical example of German nominal compounding. [https://en.wikipedia.org/wiki/Donaudampfschiffahrtsgesellschaft](https://en.wikipedia.org/wiki/Donaudampfschiffahrtsgesellschaft)[↩](https://gist.github.com/starred.atom#user-content-fnref-compound-310cd646f058931c1f7e1f9c14b67030) - Measured 2026-07-12 with `validate_lwc.py` on 8,000-character truncations of Wikipedia articles and open-source code. GPT-5.6 = OpenAI chat completions API`usage.prompt_tokens` (the three GPT-5.6 variants — luna, sol, terra — return identical counts, matching tiktoken’s o200k_base within ~6 tokens of message framing); Claude = Anthropic`count_tokens` API (Claude Opus 4.8, Fable 5, and Sonnet 5 return identical counts — one shared tokenizer); DeepSeek = OpenAI-compatible completions API (deepseek-v4-pro and v4-flash identical). An earlier five-provider run that included Gemini 3.1 Pro and Grok-4 showed the same picture (35–79% spread).[↩](https://gist.github.com/starred.atom#user-content-fnref-token-variance-310cd646f058931c1f7e1f9c14b67030) - Petrov, La Malfa, Torr, and Bibi, “Language Model Tokenizers Introduce Unfairness Between Languages,” NeurIPS 2023. arXiv:2305.15425. “The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases” (Shan vs. English under cl100k; even the cheapest languages carry a ~50% premium over English). [https://arxiv.org/abs/2305.15425](https://arxiv.org/abs/2305.15425)[↩](https://gist.github.com/starred.atom#user-content-fnref-petrov-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-petrov-2-310cd646f058931c1f7e1f9c14b67030)2[↩](https://gist.github.com/starred.atom#user-content-fnref-petrov-3-310cd646f058931c1f7e1f9c14b67030)3 - Unicode Standard Annex #11, “East Asian Width.” [https://www.unicode.org/reports/tr11/](https://www.unicode.org/reports/tr11/)[↩](https://gist.github.com/starred.atom#user-content-fnref-uax11-310cd646f058931c1f7e1f9c14b67030) - ETSI GS LIS 004 V2.0.0 (2012-07), “GMX-V: Global Information Management Metrics eXchange — Volume” (successor to the LISA standard), clause 4.2.13 “Logographic Scripts”: word counts for unspaced scripts are derived from character counts via fixed factors — Chinese 2.8, Japanese 3.0, Korean 3.3, Thai 6.0 — “based on acknowledged best practice within the Localization Industry.” [https://www.etsi.org/deliver/etsi_gs/lis/001_099/004/02.00.00_60/gs_lis004v020000p.pdf](https://www.etsi.org/deliver/etsi_gs/lis/001_099/004/02.00.00_60/gs_lis004v020000p.pdf)[↩](https://gist.github.com/starred.atom#user-content-fnref-gmxv-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-gmxv-2-310cd646f058931c1f7e1f9c14b67030)2[↩](https://gist.github.com/starred.atom#user-content-fnref-gmxv-3-310cd646f058931c1f7e1f9c14b67030)3[↩](https://gist.github.com/starred.atom#user-content-fnref-gmxv-4-310cd646f058931c1f7e1f9c14b67030)4 - E.g. AnyCount, “Word count in Oriental languages”: 1,000 Chinese characters ≈ 650–750 English words ( [https://www.anycount.com/word-count-worldwide/word-count-in-oriental-languages/](https://www.anycount.com/word-count-worldwide/word-count-in-oriental-languages/)); 1StopAsia gives the same Chinese figures and recommends character counts for Chinese/Japanese/Thai ([https://www.1stopasia.com/blog/word-count-vs-character-count-for-asian-languages/](https://www.1stopasia.com/blog/word-count-vs-character-count-for-asian-languages/)).[↩](https://gist.github.com/starred.atom#user-content-fnref-cjk-ratios-310cd646f058931c1f7e1f9c14b67030) - SWET (Society of Writers, Editors and Translators, Tokyo), “What’s in a Page?”: the 400-character genkō yōshi manuscript page corresponds to roughly 200–250 English words. [https://swet.jp/articles/article/whats_in_a_page/_C31](https://swet.jp/articles/article/whats_in_a_page/_C31)[↩](https://gist.github.com/starred.atom#user-content-fnref-swet-310cd646f058931c1f7e1f9c14b67030) - X developer documentation, “Counting characters,” and the open-source twitter-text library config (v3.json): code points U+0000–10FF plus selected general-punctuation ranges weigh 1; everything else (all CJK, Hangul, emoji) weighs 2; URLs count as 23. [https://docs.x.com/fundamentals/counting-characters](https://docs.x.com/fundamentals/counting-characters),[https://github.com/twitter/twitter-text/blob/master/config/v3.json](https://github.com/twitter/twitter-text/blob/master/config/v3.json)[↩](https://gist.github.com/starred.atom#user-content-fnref-twitter-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-twitter-2-310cd646f058931c1f7e1f9c14b67030)2 - Medium Help Center, “Read time”: “roughly 265 WPM… For posts in Chinese, Japanese and Korean, it’s a function of number of characters (500 characters/min).” Hugo ( `hugolib/page__content.go` ) uses the same 500 chars/min for CJK vs. ~213 wpm otherwise, and counts runes per CJK token when`hasCJKLanguage` is set. The`reading-time` npm package counts each CJK character as one word.[https://help.medium.com/hc/en-us/articles/214991667](https://help.medium.com/hc/en-us/articles/214991667),[https://github.com/gohugoio/hugo](https://github.com/gohugoio/hugo),[https://github.com/ngryman/reading-time](https://github.com/ngryman/reading-time)[↩](https://gist.github.com/starred.atom#user-content-fnref-medium-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-medium-2-310cd646f058931c1f7e1f9c14b67030)2 - ICU segments Chinese and Japanese with a frequency dictionary (cjdict) rather than Unicode default rules; dictionary-segmented Chinese words average roughly 1.5–2 characters. [https://unicode-org.github.io/icu/userguide/boundaryanalysis/](https://unicode-org.github.io/icu/userguide/boundaryanalysis/)[↩](https://gist.github.com/starred.atom#user-content-fnref-icu-310cd646f058931c1f7e1f9c14b67030) - Microsoft Word: “In East Asian languages, each character counts as a word” (NumChars documentation), with a separate `wdStatisticFarEastCharacters` statistic ([https://learn.microsoft.com/en-us/office/vba/api/word.wdstatistic](https://learn.microsoft.com/en-us/office/vba/api/word.wdstatistic)). memoQ: “Source words = Asian characters + non-Asian words” for Chinese/Japanese sources ([https://docs.memoq.com/current/en/Workspace/statistics.html](https://docs.memoq.com/current/en/Workspace/statistics.html)). WordPress: locale packs translate the word-count type to character counting for Japanese and Chinese (`WP_Locale::get_word_count_type()` ).[↩](https://gist.github.com/starred.atom#user-content-fnref-word-count-tools-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-word-count-tools-2-310cd646f058931c1f7e1f9c14b67030)2 - Sennrich, Haddow, and Birch, “Neural Machine Translation of Rare Words with Subword Units,” ACL 2016 — the canonical BPE-for-NLP paper. [https://aclanthology.org/P16-1162/](https://aclanthology.org/P16-1162/)[↩](https://gist.github.com/starred.atom#user-content-fnref-sennrich2016-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-sennrich2016-2-310cd646f058931c1f7e1f9c14b67030)2 - Measured on ~300k characters of English prose (Pride and Prejudice) and ~165k characters of Python stdlib source: cl100k_base 4.24/4.81 chars per token (prose/code), o200k_base 4.28/4.77, Llama 3 4.25/4.81, Gemma 2 4.29/4.03. Shorter, punctuation- or link-heavy samples run lower (3.1–3.6 in the validation corpus). [↩](https://gist.github.com/starred.atom#user-content-fnref-chars-per-token-310cd646f058931c1f7e1f9c14b67030) - OpenAI: “1 token ≈ 4 characters… ≈ ¾ of a word; 100 tokens ≈ 75 words” ( [https://help.openai.com/en/articles/4936856](https://help.openai.com/en/articles/4936856)) and “1 token is approximately 4 characters or 0.75 words for English text” ([https://developers.openai.com/api/docs/concepts](https://developers.openai.com/api/docs/concepts)). Anthropic: “a token approximately represents 3.5 English characters” ([https://platform.claude.com/docs/en/docs/resources/glossary](https://platform.claude.com/docs/en/docs/resources/glossary)). Google: “a token is equivalent to about 4 characters… 100 tokens is equal to about 60-80 English words” ([https://ai.google.dev/gemini-api/docs/tokens](https://ai.google.dev/gemini-api/docs/tokens)).[↩](https://gist.github.com/starred.atom#user-content-fnref-vendor-rules-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-vendor-rules-2-310cd646f058931c1f7e1f9c14b67030)2 - Ahia et al., “Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models,” EMNLP 2023. arXiv:2305.13707. Documents ~5× token-count and API-cost premiums for non-Latin-script languages. [https://aclanthology.org/2023.emnlp-main.614/](https://aclanthology.org/2023.emnlp-main.614/)[↩](https://gist.github.com/starred.atom#user-content-fnref-ahia-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-ahia-2-310cd646f058931c1f7e1f9c14b67030)2 - Words-per-minute measurement standardizes a “word” as 5 characters or keystrokes (including spaces and punctuation). [https://en.wikipedia.org/wiki/Words_per_minute](https://en.wikipedia.org/wiki/Words_per_minute)[↩](https://gist.github.com/starred.atom#user-content-fnref-wpm-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-wpm-2-310cd646f058931c1f7e1f9c14b67030)2 - § 11 JVEG (Justizvergütungs- und -entschädigungsgesetz): translation honoraria “für jeweils angefangene 55 Anschläge des schriftlichen Textes” (per commenced 55 keystrokes). [https://www.gesetze-im-internet.de/jveg/__11.html](https://www.gesetze-im-internet.de/jveg/__11.html). Normseite: 30 lines × 60 keystrokes = 1,800 characters (publishing convention); VG Wort uses 1,500.[https://de.wikipedia.org/wiki/Normseite](https://de.wikipedia.org/wiki/Normseite)[↩](https://gist.github.com/starred.atom#user-content-fnref-jveg-310cd646f058931c1f7e1f9c14b67030) - Amazon KDP, “Royalties in Kindle Unlimited”: KENPC (Kindle Edition Normalized Page Count) normalizes page count using standard formatting settings; the formula is unpublished (author analyses estimate ~170–200 words per page). [https://kdp.amazon.com/en_US/help/topic/G201541130](https://kdp.amazon.com/en_US/help/topic/G201541130)[↩](https://gist.github.com/starred.atom#user-content-fnref-kenpc-310cd646f058931c1f7e1f9c14b67030) - Senter and Smith, “Automated Readability Index,” AMRL-TR-66-220, 1967. Formula: ARI = 4.71 × (chars/words) + 0.5 × (words/sentences) − 21.43. [https://apps.dtic.mil/sti/tr/pdf/AD0667273.pdf](https://apps.dtic.mil/sti/tr/pdf/AD0667273.pdf)[↩](https://gist.github.com/starred.atom#user-content-fnref-ari-310cd646f058931c1f7e1f9c14b67030) - Ács, “Exploring BERT’s Vocabulary” (2019) defined subword fertility — average subwords per word — borrowing the term from statistical MT (Brown et al. 1993); Rust et al., “How Good is Your Tokenizer?” (ACL 2021) established it as the standard tokenizer-efficiency metric. [https://juditacs.github.io/2019/02/19/bert-tokenization-stats.html](https://juditacs.github.io/2019/02/19/bert-tokenization-stats.html),[https://aclanthology.org/2021.acl-long.243/](https://aclanthology.org/2021.acl-long.243/)[↩](https://gist.github.com/starred.atom#user-content-fnref-fertility-310cd646f058931c1f7e1f9c14b67030) - tokenx: character-ratio estimation with per-language overrides, “~96% accuracy” ( [https://github.com/johannschopplich/tokenx](https://github.com/johannschopplich/tokenx)). LangChain`count_tokens_approximately` : chars ÷ 4 plus per-message overhead. Microsoft Semantic Kernel TextChunker: length ÷ 4.[↩](https://gist.github.com/starred.atom#user-content-fnref-estimators-310cd646f058931c1f7e1f9c14b67030)