Logical word count: a cheap, tokenizer-free measure of text length that tracks LLM token counts across languages, code, and mixed content

A developer introduced Logical Word Count, a cheap, tokenizer-free measure of text length that stays roughly proportional to LLM token counts across languages, code, and mixed content. The method uses two simple rules: wide characters count as half a logical word, and whitespace-split words are clamped to an average length between 3 and 6 characters. Empirical tests show tokens per logical word vary within a ~2× band across tokenizers, compared to a 24× spread for literal word count.

Logical word count is a cheap, deterministic, tokenizer-free measure of text length that stays roughly proportional to LLM token counts across natural languages, code, and mixed content — while remaining equal to ordinary word count for plain English prose. It is defined in two rules: Wide East Asian characters count as ½ logical word each. This covers Chinese characters, Japanese kana, Korean Hangul, and CJK punctuation. Everything else is split on whitespace, and the word count is clamped so the average word length stays between 3 and 6 characters. Text with very long “words” compounds, identifiers, URLs, minified code is counted as chars / 6 ; text with very short words is counted as chars / 3 ; typical prose passes through unchanged. A reference implementation is ~15 lines with no dependencies see Reference implementation https://gist.github.com/starred.atom reference-implementation . Empirically, tokens per logical word stays within a ~2× band for every tokenizer tested — 1.36–2.24 for the GPT-5.6 / o200k family — across ten languages plus source code and machine formats, versus a 24× spread for literal word count and a 3× spread for the common chars ÷ 4 rule on the same corpus see Validation https://gist.github.com/starred.atom validation . Word count is the standard unit for measuring text length, but it breaks in two distinct ways: it can under-count content volume, and it can over-count it. CJK text has few or no word-delimiting spaces; Unicode’s own segmentation standard notes that word boundaries in Chinese, Japanese, or Thai require dictionary lookup. 1 A 500-character Chinese paragraph counts as a handful of whitespace-delimited “words” despite carrying the information of roughly 250 English words. In our test corpus, an 8,000-character Chinese Wikipedia extract contains just 165 whitespace-delimited tokens. German compounds like Donaudampfschifffahrtsgesellschaft 2 pack five lexical units into one “word”; agglutinative languages like Finnish average 8+ characters per whitespace-delimited word versus ~5.5 for English. Identical content, materially different word counts. Code is the hardest case, for two reasons: Symbols are rarely space-separated. The expression a+b clamp x,lo,hi is one whitespace-delimited “word” but contains eight semantic tokens. Minified JavaScript averages 44 characters per whitespace-delimited “word” in our corpus; CSV rows and JSON average 10–14. Identifiers vary wildly in length. x and calculateAverageResponseTime are both one word, but the latter contains five semantic units. Code with long descriptive names is under-counted relative to its actual volume. Documents that combine prose, code blocks, tables, links, and CJK text have no single word-counting strategy that works across all sections. A README with badges, URLs, and code samples measures 11.7 characters per “word” — double typical prose. Token counts solve the normalization problem by construction, but they are not stable across model providers, they require a tokenizer vocabulary, and they are orders of magnitude more expensive to compute than a character scan. Token counts vary across providers — and across model generations within a provider. The same 8,000-character samples measured against the latest production tokenizers July 2026 : 3 https://gist.github.com/starred.atom user-content-fn-token-variance-310cd646f058931c1f7e1f9c14b67030 | Sample | GPT-5.6 | Claude Opus 4.8 / Fable 5 | DeepSeek V4 | max deviation | |---|---|---|---|---| | English Wikipedia | 1,657 | 2,471 | 1,664 | 49% | | German Wikipedia | 1,816 | 3,695 | 2,068 | 103% | | Chinese Wikipedia | 5,351 | 7,348 | 4,333 | 70% | | Japanese Wikipedia | 5,097 | 6,403 | 4,343 | 47% | | Korean Wikipedia | 4,529 | 7,916 | 5,026 | 75% | | Thai Wikipedia | 3,520 | 7,107 | 3,238 | 119% | | Python code | 1,853 | 2,810 | 1,893 | 52% | | TypeScript code | 1,907 | 3,518 | 2,004 | 84% | Even for English the tokenizers diverge by 49%, and for German by 103%. Nor are counts stable over time: Anthropic’s newest tokenizer shared by Claude Opus 4.8, Fable 5, and Sonnet 5 spends ~37% more tokens on the same English text than the Opus 4.6 tokenizer did, and ~2× on Thai. Academic work documents far larger gaps across languages: the same content can take up to 15× more tokens in some languages than in English. 4 A “token count” is therefore not a property of a text — it is a property of a text, model pair. Any model-independent measure has this cross-tokenizer variance as a floor on its achievable accuracy, and that is fine: the goal is a stable measure of content volume that tokens track within a predictable band, not a replacement for a tokenizer when exact counts matter. Token counts also require a vocabulary. Computing them means shipping or calling the target model’s tokenizer — a dependency that may not exist where length is measured a database check constraint, a CLI, an editor plugin, a non-Python runtime and that changes when models change. Given a text string: Wide characters. Count the characters whose Unicode East Asian Width is Wide or Fullwidth— Han ideographs, Hiragana, Katakana, Hangul syllables, CJK punctuation, and fullwidth forms. Call this 5 https://gist.github.com/starred.atom user-content-fn-uax11-310cd646f058931c1f7e1f9c14b67030 wide . Each contributes ½ logical word. For splitting purposes, treat each wide character as if it were surrounded by spaces. Remaining text. Split what remains on whitespace. Let words be the number of resulting tokens and chars the total count of their characters Unicode code points . Clamp the word count so that average word length stays in L, U = 3, 6 : clamped = clamp words, chars / 6, chars / 3 = max chars / 6, min chars / 3, words logical words = round wide / 2 + clamped Empty or whitespace-only text has 0 logical words. The clamp activates only when average characters per word falls outside 3, 6 . Most English and European prose runs 4.5–5.5 characters per whitespace-delimited word punctuation attached , so for ordinary prose logical word count equals literal word count — the measure is backward-compatible where word count already works. Long-word text German and Finnish compounds, identifiers, URLs, minified code, CSV rows is counted as chars / 6 ; degenerate short-word text e.g. tables of single digits is capped at chars / 3 . In practice the upper bound does nearly all the work; the lower bound is a safety rail that real content rarely hits. Canonical definition: East Asian Width ∈ {W, F} available in Python’s unicodedata , ICU, and most Unicode libraries . Where the property is unavailable, a range-based approximation — the Han and Hangul scripts plus the CJK punctuation-through-Katakana blocks U+3000–30FF , Katakana phonetic extensions U+31F0–31FF , and fullwidth forms U+FF00–FFEF — produces identical counts on all our CJK test samples kana must be matched by block, not script property: ー U+30FC and ・ U+30FB are Script=Common but Wide . Rare code points halfwidth Katakana, some symbols may still differ between the two definitions; the effect on real text is well under 1%. Korean: Hangul is space-delimited, but each syllable block encodes 2–3 jamo and tokenizers price Korean like Chinese and Japanese ~1.4 chars/token . Weighting Hangul syllables at ½ overrides the space-based count and empirically aligns Korean with every other language K = 1.57 vs. 2.54 without the rule; see Validation https://gist.github.com/starred.atom validation . The localization industry’s GMX-V standard likewise groups Korean with the logographic scripts. 6 https://gist.github.com/starred.atom user-content-fn-gmxv-310cd646f058931c1f7e1f9c14b67030 Emoji are East Asian Width Wide, so they count ½ each. Tokenizers typically spend 1–3 tokens per emoji; at realistic emoji densities the error is negligible. Independent conventions, developed for different purposes, converge on roughly two CJK characters per English-word-equivalent: | Convention | Effective rate | Purpose | |---|---|---| | Translation-industry rule of thumb | ~1,000 Japanese chars ≈ 400–500 English words; 1,000 Chinese chars ≈ 650–750 English words | Quoting/billing | | Japanese 原稿用紙 genkō yōshi manuscript page | 400 chars ≈ 200–250 English words | Publishing | | Twitter/X character limit | CJK weighted 2× 280-unit limit = 140 CJK chars | Length limiting | | Medium & Hugo reading speed | 500 CJK chars/min vs. ~212–265 words/min ≈ 2–2.4 chars per word | Reading time | | ICU dictionary segmentation | Chinese words average ~1.5–2 chars | Linguistic segmentation | | Modern LLM tokenizers measured | one CJK char ≈ 0.7 tokens ≈ half the ~1.35 tokens of an English word | Token accounting | The tokenizer alignment is the one this spec is calibrated to: weighting CJK characters at ½ word puts Chinese, Japanese, and Korean at 1.54–1.57 tokens per logical word under o200k — statistically indistinguishable from French 1.50 and German 1.57 . GMX-V, the localization industry’s published volume standard, uses somewhat higher divisors for source-word counts Chinese ÷ 2.8, Japanese ÷ 3.0, Korean ÷ 3.3 6 — it estimates the number of words a native segmentation would find, whereas logical word count like the billing ratios above targets volume equivalence. Simple word-counting tools mostly land on 1× instead: Microsoft Word and memoQ count each CJK character as one word, and Unicode’s default word segmentation breaks between every pair of ideographs. 12 https://gist.github.com/starred.atom user-content-fn-word-count-tools-310cd646f058931c1f7e1f9c14b67030 Half a word per character sits deliberately between “one word per character” too high: overweights CJK ~2× in any cross-language comparison and GMX-V’s ~⅓ too low for token tracking . 1 https://gist.github.com/starred.atom user-content-fn-uax29-310cd646f058931c1f7e1f9c14b67030 BPE tokenizers merge frequent byte sequences until each token carries roughly one “meaning-sized” chunk. 13 Measured on large samples, modern large-vocabulary tokenizers o200k, cl100k, Llama 3, Gemma 2 all land near 4.2–4.3 characters per token for English prose and 4.0–4.8 for source code; vendor rules of thumb agree OpenAI: ~4 chars or ¾ of a word per token; Anthropic: ~3.5 chars; Google: ~4 chars . 14 https://gist.github.com/starred.atom user-content-fn-chars-per-token-310cd646f058931c1f7e1f9c14b67030 Anthropic’s newest tokenizer is denser — ~2.7 characters per token on our English sample — but uniformly so across content types, which shifts K by a constant factor without widening the band the clamp is designed to hold. 15 https://gist.github.com/starred.atom user-content-fn-vendor-rules-310cd646f058931c1f7e1f9c14b67030 The bounds bracket that range without a vocabulary table: U = 6 is the point where long-word content compounds, identifiers, URLs stops gaining count from packing more characters into each word. chars / 6 tracks the ~4–5 chars/token that tokenizers actually spend on such content, at a stable ratio of ~1.3–1.5 tokens per logical word. L = 3 caps degenerate short-word content. Real prose almost never averages below 3 even Korean, with dense syllable blocks, averages 3.5 ; the bound exists so that pathological inputs single-character tables, heavily spaced tokens cannot inflate the count more than ~1.5× over prose.- Everything between 3 and 6 chars/word passes through, preserving logical words = literal words for ordinary prose in English, French, Spanish, Russian, Hebrew, and most other spaced languages. Tightening the bounds e.g. 3, 5 narrows the token-tracking band another ~10% but sacrifices exact backward compatibility with English word count; loosening them e.g. 2, 7 admits outliers without benefit. 3, 6 with weight ½ is the recommended convention; all three constants are parameters in the reference implementation for applications that want to recalibrate. Python canonical : python import unicodedata def logical word count text: str, lower: float = 3.0, upper: float = 6.0, wide weight: float = 0.5 - int: wide = 0 rest = for c in text: if unicodedata.east asian width c in "W", "F" : wide += 1 rest.append " " else: rest.append c words = "".join rest .split chars = sum len w for w in words clamped = max chars / upper, min chars / lower, len words return round wide wide weight + clamped TypeScript script-range approximation of East Asian Width; identical results on real-world text : js const WIDE = / \p{Script=Han}\p{Script=Hangul}\u3000-\u30FF\u31F0-\u31FF\uFF00-\uFFEF /u; function logicalWordCount text: string, lower = 3, upper = 6, wideWeight = 0.5, : number { let wide = 0; const rest: string = ; for const ch of text { // iterates code points, not UTF-16 units if WIDE.test ch { wide++; rest.push " " ; } else { rest.push ch ; } } const words = rest.join "" .split /\s+/ .filter Boolean ; const chars = words.reduce n, w = n + ...w .length, 0 ; const clamped = Math.max chars / upper, Math.min chars / lower, words.length ; return Math.round wide wideWeight + clamped ; } Both are O n , allocation-light, and dependency-free. Measured with the companion script validate lwc.py on 8,000-character samples: Wikipedia extracts in ten languages Hebrew and Thai are reported under Limitations https://gist.github.com/starred.atom limitations , real source files, minified JS, CSV, JSON, and a mixed-content README. Token counts are GPT-5.6 via the OpenAI API the o200k tokenizer family ; K = tokens / logical words. | Sample | chars | literal words | logical words | tokens | K | |---|---|---|---|---|---| | English Wikipedia | 6,775 | 1,219 | 1,219 | 1,657 | 1.36 | | French Wikipedia | 6,756 | 1,222 | 1,222 | 1,838 | 1.50 | | German Wikipedia | 6,921 | 1,063 | 1,154 | 1,816 | 1.57 | | Russian Wikipedia | 6,998 | 995 | 1,166 | 1,939 | 1.66 | | Finnish Wikipedia | 7,104 | 868 | 1,184 | 2,309 | 1.95 | | Chinese Wikipedia | 7,807 | 165 | 3,400 | 5,351 | 1.57 | | Japanese Wikipedia | 7,642 | 300 | 3,300 | 5,097 | 1.54 | | Korean Wikipedia | 6,181 | 1,781 | 2,888 | 4,529 | 1.57 | | Python code | 5,709 | 862 | 952 | 1,853 | 1.95 | | TypeScript code | 6,800 | 865 | 1,133 | 1,907 | 1.68 | | Markdown README | 7,314 | 627 | 1,219 | 2,178 | 1.79 | | Minified JavaScript | 7,825 | 176 | 1,304 | 2,889 | 2.22 | | CSV data | 7,462 | 539 | 1,244 | 2,786 | 2.24 | | JSON | 7,037 | 560 | 1,173 | 2,168 | 1.85 | Prose, code, and CJK all land in K = 1.36–1.95 a 1.43× band . Punctuation-dense machine formats run hotter, up to ~2.2. Compare the alternatives on the identical corpus: | Measure | tokens ÷ measure, across samples | spread | |---|---|---| logical words | 1.36 – 2.24 | 1.65× | | literal words | 1.36 – 32.4 | 24× | | chars ÷ 4 | 0.98 – 2.9 | 3.0× | The same holds for the other tokenizer families. On the identical corpus including the Hebrew and Thai limitation cases , tokens per logical word spans a 2.0× band for GPT-5.6, 2.1× for DeepSeek V4, and 2.9× for the Claude Opus 4.8 / Fable 5 tokenizer — each centered on its own K — while literal word count spans 19–24× on every one of them. The remaining variation is largely irreducible: the tokenizers themselves disagree by 37–119% on identical text see the table in Motivation https://gist.github.com/starred.atom why-not-just-use-token-counts , so no model-free measure can track any single model much more tightly than models track each other. For a quick estimate against o200k-family tokenizers GPT-5.x; DeepSeek V4 runs ~10% higher : estimated tokens ≈ logical words × 1.6 This lands within roughly ±25% across prose any script , code, and markdown. Useful refinements: Clean English prose: use ×1.35 — which is the familiar “1 word ≈ 4⁄3 tokens” vendor rule,recovered exactly because logical words equal literal words for such text. 15 https://gist.github.com/starred.atom user-content-fn-vendor-rules-310cd646f058931c1f7e1f9c14b67030 Punctuation-dense machine formats minified code, CSV, logs : use ×2.2, or accept under-estimation. Model-specific correction: apply a per-tokenizer factor — DeepSeek V4 runs ~1.1× o200k, and Anthropic’s newest tokenizer Claude Opus 4.8 / Fable 5 / Sonnet 5 runs ~1.5–2× o200k depending on content, i.e. tokens ≈ logical words × 2.75 overall. This correction is roughly constant per model, unlike the correction needed for raw word count, which varies by an order of magnitude with text type. When exact counts matter billing, hard context limits , use the model’s tokenizer; logical word count is for the much larger class of uses where a stable, cheap, model-free number is worth ±25%. Cheap: O n single pass, no vocabulary, no dependencies. Deterministic and model-free: same input, same count, forever. Backward-compatible: equals literal word count for ordinary English and most European-language prose. Script-aware but language-blind: uses only character properties; no language detection, dictionaries, or segmentation models. Monotonic: appending non-whitespace content never decreases the count. Additive: splitting text at whitespace boundaries and summing the parts’ counts approximates the whole within rounding , so counts can be computed incrementally or in parallel. Bounded token ratio: every tokenizer tested spends a near-constant number of tokens per logical word — within a ~2–3× band across all content types, versus ~20–24× for literal word count. Stated plainly, since a measure is only trustworthy if its failure modes are known: Unspaced non-CJK scripts Thai, Lao, Khmer, Myanmar are not covered by the wide-character rule; they fall to the clamp’s chars / 6 floor. That happens to match GMX-V’s convention for Thai chars ÷ 6 ,but o200k still spends ~2.75 tokens per logical word on Thai Anthropic’s newest tokenizer, ~5.6 — high fertility for these scripts is a property of the tokenizers, not of the measure. Applications centered on these scripts should add a per-script weight Thai ≈ ¼–⅓ word/char by the same method used here for CJK. 6 https://gist.github.com/starred.atom user-content-fn-gmxv-310cd646f058931c1f7e1f9c14b67030 Non-Latin alphabetic scripts Hebrew, Greek, Devanagari, … count correctly as volume — their word counts are true word counts — but tokenizers charge them 1.9–2.3 tokens per logical word versus 1.35 for English o200k; other tokenizer families show the same skew at their own scale . This is the well-documented tokenizer-fairness gap, 4 https://gist.github.com/starred.atom user-content-fn-petrov-310cd646f058931c1f7e1f9c14b67030 and logical word count deliberately does not bend the volume measure to compensate for it. 16 https://gist.github.com/starred.atom user-content-fn-ahia-310cd646f058931c1f7e1f9c14b67030 Machine formats minified code, CSV, dense JSON run ~1.4× hotter in tokens than prose. The clamp keeps them bounded literal word count is off by 24× here , but a single multiplier will under-estimate their token cost. It is not a linguistic word count. For Chinese, ½ word per character approximates dictionary segmentation Chinese words average 1.5–2 characters , but no claim is made that logical words matches what a human or a segmenter would count as words in any specific language. It is a normalized volume unit, like a “standard page” — not a lexical analysis. No existing term or formula was found for this specific construction a chars-per-word clamp plus a wide-character weight , and the name “logical word count” appears to be unclaimed. "Standard word" already means 5 keystrokes in typing measurement; 17 “weighted word count” means fuzzy-match discounts in CAT tools. The pieces, however, have deep roots: GMX-V LISA → ETSI GS LIS 004 — the localization industry’s word/character count standard — is the closest precedent. Version 1.0 2007 refused to segment unspaced scripts and required character counts; version 2.0 2012 added normative per-script divisors to convert characters to words: Chinese ÷ 2.8, Japanese ÷ 3.0, Korean ÷ 3.3, Thai ÷ 6, described as “acknowledged best practice within the Localization Industry.”Logical word count generalizes the same move — chars ÷ factor — and extends it to code and mixed content via the clamp. 6 https://gist.github.com/starred.atom user-content-fn-gmxv-310cd646f058931c1f7e1f9c14b67030 CJK character-as-word conventions: Microsoft Word’s “Words” statistic counts each East Asian character as a word with a separate Asian-characters statistic ; memoQ computes source words as Asian characters + non-Asian words; WordPress locale packs switch Japanese and Chinese to character counting outright. 12 https://gist.github.com/starred.atom user-content-fn-word-count-tools-310cd646f058931c1f7e1f9c14b67030 Weighted length limits: Twitter/X’s 280-unit limit weights code points beyond a small Latin-ish set roughly through U+10FF plus general punctuation at 2, so CJK posts max out at 140 characters — a shipped, planet-scale precedent for weighting wide characters at exactly 2× Latin. 9 https://gist.github.com/starred.atom user-content-fn-twitter-310cd646f058931c1f7e1f9c14b67030 Reading-time estimators: Medium uses ~265 words/min for spaced languages and 500 characters/min for CJK; Hugo and the reading-time npm package used by Gatsby, Ghost analogues count CJK per-character against a ~2.4× higher rate. 10 https://gist.github.com/starred.atom user-content-fn-medium-310cd646f058931c1f7e1f9c14b67030 Character-based billing units: Germany’s court-interpreting law § 11 JVEG prices translation per 55 keystrokes "je angefangene 55 Anschläge" ; publishing standard pages Normseite: 1,500–1,800 chars and typing’s 5-character “standard word” all define length by characters when word counts are unreliable or unfair. 18 https://gist.github.com/starred.atom user-content-fn-jveg-310cd646f058931c1f7e1f9c14b67030 Amazon’s KENPC normalizes e-book length to a synthetic page unit for paying authors. 17 https://gist.github.com/starred.atom user-content-fn-wpm-310cd646f058931c1f7e1f9c14b67030 19 https://gist.github.com/starred.atom user-content-fn-kenpc-310cd646f058931c1f7e1f9c14b67030 Character length as a text statistic: the Automated Readability Index 1967 used chars/word as a machine-friendly complexity proxy precisely because characters were cheap to count. 20 https://gist.github.com/starred.atom user-content-fn-ari-310cd646f058931c1f7e1f9c14b67030 Tokenizer research: subword fertility tokens per word is the standard metric for tokenizer efficiency;the unfairness of token counts across languages is documented up to 15×. 21 https://gist.github.com/starred.atom user-content-fn-fertility-310cd646f058931c1f7e1f9c14b67030 4 https://gist.github.com/starred.atom user-content-fn-petrov-310cd646f058931c1f7e1f9c14b67030 BPE itself 16 https://gist.github.com/starred.atom user-content-fn-ahia-310cd646f058931c1f7e1f9c14b67030 solves normalization by construction, at the cost of a vocabulary and model dependence. 13 https://gist.github.com/starred.atom user-content-fn-sennrich2016-310cd646f058931c1f7e1f9c14b67030 Tokenizer-free token estimators the tokenx npm package; LangChain’s count tokens approximately ; Semantic Kernel’s chars ÷ 4 estimate tokens for a specific model family from character ratios.Logical word count differs in intent: it is a stable text measure first — human-legible, word-count compatible — that happens to track tokens within a known band, rather than a model-specific predictor. 22 https://gist.github.com/starred.atom user-content-fn-estimators-310cd646f058931c1f7e1f9c14b67030 All measurements in this document come from the companion script validate lwc.py published alongside this document , which fetches the public samples, computes literal words, logical words, and exact token counts via provider APIs, and prints the comparison tables: uv run validate lwc.py o200k locally; other providers with API keys uv run validate lwc.py --lower 3 --upper 6 --wide-weight 0.5 uv run validate lwc.py --list-samples With no API keys, OpenAI counts fall back to local tiktoken o200k base — measured identical to the API counts modulo ~6 tokens of message framing . Set OPENAI API KEY , ANTHROPIC API KEY , and/or DEEPSEEK API KEY to reproduce the cross-model table with exact provider counts; Gemini and xAI models are supported via --models with their keys. The script’s dependencies are pinned under a two-week cooling-off policy uv’s exclude-newer . Footnotes - Unicode Standard Annex 29, “Unicode Text Segmentation,” §4 Word Boundaries: default rules break between every pair of ideographs rule WB999 , and “reliable detection of word boundaries in languages such as Thai, Lao, Chinese, or Japanese requires the use of dictionary lookup or other mechanisms.” https://unicode.org/reports/tr29/ https://unicode.org/reports/tr29/ ↩ https://gist.github.com/starred.atom user-content-fnref-uax29-310cd646f058931c1f7e1f9c14b67030 ↩ https://gist.github.com/starred.atom user-content-fnref-uax29-2-310cd646f058931c1f7e1f9c14b67030 2 - Danube–steam–ship–journey–company; the Erste Donau-Dampfschiffahrts-Gesellschaft founded 1829 is the canonical example of German nominal compounding. https://en.wikipedia.org/wiki/Donaudampfschiffahrtsgesellschaft https://en.wikipedia.org/wiki/Donaudampfschiffahrtsgesellschaft ↩ https://gist.github.com/starred.atom user-content-fnref-compound-310cd646f058931c1f7e1f9c14b67030 - Measured 2026-07-12 with validate lwc.py on 8,000-character truncations of Wikipedia articles and open-source code. GPT-5.6 = OpenAI chat completions API usage.prompt tokens the three GPT-5.6 variants — luna, sol, terra — return identical counts, matching tiktoken’s o200k base within ~6 tokens of message framing ; Claude = Anthropic count tokens API Claude Opus 4.8, Fable 5, and Sonnet 5 return identical counts — one shared tokenizer ; DeepSeek = OpenAI-compatible completions API deepseek-v4-pro and v4-flash identical . An earlier five-provider run that included Gemini 3.1 Pro and Grok-4 showed the same picture 35–79% spread . ↩ https://gist.github.com/starred.atom user-content-fnref-token-variance-310cd646f058931c1f7e1f9c14b67030 - Petrov, La Malfa, Torr, and Bibi, “Language Model Tokenizers Introduce Unfairness Between Languages,” NeurIPS 2023. arXiv:2305.15425. “The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases” Shan vs. English under cl100k; even the cheapest languages carry a ~50% premium over English . https://arxiv.org/abs/2305.15425 https://arxiv.org/abs/2305.15425 ↩ https://gist.github.com/starred.atom user-content-fnref-petrov-310cd646f058931c1f7e1f9c14b67030 ↩ https://gist.github.com/starred.atom user-content-fnref-petrov-2-310cd646f058931c1f7e1f9c14b67030 2 ↩ https://gist.github.com/starred.atom user-content-fnref-petrov-3-310cd646f058931c1f7e1f9c14b67030 3 - Unicode Standard Annex 11, “East Asian Width.” https://www.unicode.org/reports/tr11/ https://www.unicode.org/reports/tr11/ ↩ https://gist.github.com/starred.atom user-content-fnref-uax11-310cd646f058931c1f7e1f9c14b67030 - ETSI GS LIS 004 V2.0.0 2012-07 , “GMX-V: Global Information Management Metrics eXchange — Volume” successor to the LISA standard , clause 4.2.13 “Logographic Scripts”: word counts for unspaced scripts are derived from character counts via fixed factors — Chinese 2.8, Japanese 3.0, Korean 3.3, Thai 6.0 — “based on acknowledged best practice within the Localization Industry.” https://www.etsi.org/deliver/etsi gs/lis/001 099/004/02.00.00 60/gs lis004v020000p.pdf https://www.etsi.org/deliver/etsi gs/lis/001 099/004/02.00.00 60/gs lis004v020000p.pdf ↩ https://gist.github.com/starred.atom user-content-fnref-gmxv-310cd646f058931c1f7e1f9c14b67030 ↩ https://gist.github.com/starred.atom user-content-fnref-gmxv-2-310cd646f058931c1f7e1f9c14b67030 2 ↩ https://gist.github.com/starred.atom user-content-fnref-gmxv-3-310cd646f058931c1f7e1f9c14b67030 3 ↩ https://gist.github.com/starred.atom user-content-fnref-gmxv-4-310cd646f058931c1f7e1f9c14b67030 4 - E.g. AnyCount, “Word count in Oriental languages”: 1,000 Chinese characters ≈ 650–750 English words https://www.anycount.com/word-count-worldwide/word-count-in-oriental-languages/ https://www.anycount.com/word-count-worldwide/word-count-in-oriental-languages/ ; 1StopAsia gives the same Chinese figures and recommends character counts for Chinese/Japanese/Thai https://www.1stopasia.com/blog/word-count-vs-character-count-for-asian-languages/ https://www.1stopasia.com/blog/word-count-vs-character-count-for-asian-languages/ . ↩ https://gist.github.com/starred.atom user-content-fnref-cjk-ratios-310cd646f058931c1f7e1f9c14b67030 - SWET Society of Writers, Editors and Translators, Tokyo , “What’s in a Page?”: the 400-character genkō yōshi manuscript page corresponds to roughly 200–250 English words. https://swet.jp/articles/article/whats in a page/ C31 https://swet.jp/articles/article/whats in a page/ C31 ↩ https://gist.github.com/starred.atom user-content-fnref-swet-310cd646f058931c1f7e1f9c14b67030 - X developer documentation, “Counting characters,” and the open-source twitter-text library config v3.json : code points U+0000–10FF plus selected general-punctuation ranges weigh 1; everything else all CJK, Hangul, emoji weighs 2; URLs count as 23. https://docs.x.com/fundamentals/counting-characters https://docs.x.com/fundamentals/counting-characters , https://github.com/twitter/twitter-text/blob/master/config/v3.json https://github.com/twitter/twitter-text/blob/master/config/v3.json ↩ https://gist.github.com/starred.atom user-content-fnref-twitter-310cd646f058931c1f7e1f9c14b67030 ↩ https://gist.github.com/starred.atom user-content-fnref-twitter-2-310cd646f058931c1f7e1f9c14b67030 2 - Medium Help Center, “Read time”: “roughly 265 WPM… For posts in Chinese, Japanese and Korean, it’s a function of number of characters 500 characters/min .” Hugo hugolib/page content.go uses the same 500 chars/min for CJK vs. ~213 wpm otherwise, and counts runes per CJK token when hasCJKLanguage is set. The reading-time npm package counts each CJK character as one word. https://help.medium.com/hc/en-us/articles/214991667 https://help.medium.com/hc/en-us/articles/214991667 , https://github.com/gohugoio/hugo https://github.com/gohugoio/hugo , https://github.com/ngryman/reading-time https://github.com/ngryman/reading-time ↩ https://gist.github.com/starred.atom user-content-fnref-medium-310cd646f058931c1f7e1f9c14b67030 ↩ https://gist.github.com/starred.atom user-content-fnref-medium-2-310cd646f058931c1f7e1f9c14b67030 2 - ICU segments Chinese and Japanese with a frequency dictionary cjdict rather than Unicode default rules; dictionary-segmented Chinese words average roughly 1.5–2 characters. https://unicode-org.github.io/icu/userguide/boundaryanalysis/ https://unicode-org.github.io/icu/userguide/boundaryanalysis/ ↩ https://gist.github.com/starred.atom user-content-fnref-icu-310cd646f058931c1f7e1f9c14b67030 - Microsoft Word: “In East Asian languages, each character counts as a word” NumChars documentation , with a separate wdStatisticFarEastCharacters statistic https://learn.microsoft.com/en-us/office/vba/api/word.wdstatistic https://learn.microsoft.com/en-us/office/vba/api/word.wdstatistic . memoQ: “Source words = Asian characters + non-Asian words” for Chinese/Japanese sources https://docs.memoq.com/current/en/Workspace/statistics.html https://docs.memoq.com/current/en/Workspace/statistics.html . WordPress: locale packs translate the word-count type to character counting for Japanese and Chinese WP Locale::get word count type . ↩ https://gist.github.com/starred.atom user-content-fnref-word-count-tools-310cd646f058931c1f7e1f9c14b67030 ↩ https://gist.github.com/starred.atom user-content-fnref-word-count-tools-2-310cd646f058931c1f7e1f9c14b67030 2 - Sennrich, Haddow, and Birch, “Neural Machine Translation of Rare Words with Subword Units,” ACL 2016 — the canonical BPE-for-NLP paper. https://aclanthology.org/P16-1162/ https://aclanthology.org/P16-1162/ ↩ https://gist.github.com/starred.atom user-content-fnref-sennrich2016-310cd646f058931c1f7e1f9c14b67030 ↩ https://gist.github.com/starred.atom user-content-fnref-sennrich2016-2-310cd646f058931c1f7e1f9c14b67030 2 - Measured on ~300k characters of English prose Pride and Prejudice and ~165k characters of Python stdlib source: cl100k base 4.24/4.81 chars per token prose/code , o200k base 4.28/4.77, Llama 3 4.25/4.81, Gemma 2 4.29/4.03. Shorter, punctuation- or link-heavy samples run lower 3.1–3.6 in the validation corpus . ↩ https://gist.github.com/starred.atom user-content-fnref-chars-per-token-310cd646f058931c1f7e1f9c14b67030 - OpenAI: “1 token ≈ 4 characters… ≈ ¾ of a word; 100 tokens ≈ 75 words” https://help.openai.com/en/articles/4936856 https://help.openai.com/en/articles/4936856 and “1 token is approximately 4 characters or 0.75 words for English text” https://developers.openai.com/api/docs/concepts https://developers.openai.com/api/docs/concepts . Anthropic: “a token approximately represents 3.5 English characters” https://platform.claude.com/docs/en/docs/resources/glossary https://platform.claude.com/docs/en/docs/resources/glossary . Google: “a token is equivalent to about 4 characters… 100 tokens is equal to about 60-80 English words” https://ai.google.dev/gemini-api/docs/tokens https://ai.google.dev/gemini-api/docs/tokens . ↩ https://gist.github.com/starred.atom user-content-fnref-vendor-rules-310cd646f058931c1f7e1f9c14b67030 ↩ https://gist.github.com/starred.atom user-content-fnref-vendor-rules-2-310cd646f058931c1f7e1f9c14b67030 2 - Ahia et al., “Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models,” EMNLP 2023. arXiv:2305.13707. Documents ~5× token-count and API-cost premiums for non-Latin-script languages. https://aclanthology.org/2023.emnlp-main.614/ https://aclanthology.org/2023.emnlp-main.614/ ↩ https://gist.github.com/starred.atom user-content-fnref-ahia-310cd646f058931c1f7e1f9c14b67030 ↩ https://gist.github.com/starred.atom user-content-fnref-ahia-2-310cd646f058931c1f7e1f9c14b67030 2 - Words-per-minute measurement standardizes a “word” as 5 characters or keystrokes including spaces and punctuation . https://en.wikipedia.org/wiki/Words per minute https://en.wikipedia.org/wiki/Words per minute ↩ https://gist.github.com/starred.atom user-content-fnref-wpm-310cd646f058931c1f7e1f9c14b67030 ↩ https://gist.github.com/starred.atom user-content-fnref-wpm-2-310cd646f058931c1f7e1f9c14b67030 2 - § 11 JVEG Justizvergütungs- und -entschädigungsgesetz : translation honoraria “für jeweils angefangene 55 Anschläge des schriftlichen Textes” per commenced 55 keystrokes . https://www.gesetze-im-internet.de/jveg/ 11.html https://www.gesetze-im-internet.de/jveg/ 11.html . Normseite: 30 lines × 60 keystrokes = 1,800 characters publishing convention ; VG Wort uses 1,500. https://de.wikipedia.org/wiki/Normseite https://de.wikipedia.org/wiki/Normseite ↩ https://gist.github.com/starred.atom user-content-fnref-jveg-310cd646f058931c1f7e1f9c14b67030 - Amazon KDP, “Royalties in Kindle Unlimited”: KENPC Kindle Edition Normalized Page Count normalizes page count using standard formatting settings; the formula is unpublished author analyses estimate ~170–200 words per page . https://kdp.amazon.com/en US/help/topic/G201541130 https://kdp.amazon.com/en US/help/topic/G201541130 ↩ https://gist.github.com/starred.atom user-content-fnref-kenpc-310cd646f058931c1f7e1f9c14b67030 - Senter and Smith, “Automated Readability Index,” AMRL-TR-66-220, 1967. Formula: ARI = 4.71 × chars/words + 0.5 × words/sentences − 21.43. https://apps.dtic.mil/sti/tr/pdf/AD0667273.pdf https://apps.dtic.mil/sti/tr/pdf/AD0667273.pdf ↩ https://gist.github.com/starred.atom user-content-fnref-ari-310cd646f058931c1f7e1f9c14b67030 - Ács, “Exploring BERT’s Vocabulary” 2019 defined subword fertility — average subwords per word — borrowing the term from statistical MT Brown et al. 1993 ; Rust et al., “How Good is Your Tokenizer?” ACL 2021 established it as the standard tokenizer-efficiency metric. https://juditacs.github.io/2019/02/19/bert-tokenization-stats.html https://juditacs.github.io/2019/02/19/bert-tokenization-stats.html , https://aclanthology.org/2021.acl-long.243/ https://aclanthology.org/2021.acl-long.243/ ↩ https://gist.github.com/starred.atom user-content-fnref-fertility-310cd646f058931c1f7e1f9c14b67030 - tokenx: character-ratio estimation with per-language overrides, “~96% accuracy” https://github.com/johannschopplich/tokenx https://github.com/johannschopplich/tokenx . LangChain count tokens approximately : chars ÷ 4 plus per-message overhead. Microsoft Semantic Kernel TextChunker: length ÷ 4. ↩ https://gist.github.com/starred.atom user-content-fnref-estimators-310cd646f058931c1f7e1f9c14b67030