{"slug": "logical-word-count-a-cheap-tokenizer-free-measure-of-text-length-that-tracks-llm", "title": "Logical word count: a cheap, tokenizer-free measure of text length that tracks LLM token counts across languages, code, and mixed content", "summary": "A developer introduced Logical Word Count, a cheap, tokenizer-free measure of text length that stays roughly proportional to LLM token counts across languages, code, and mixed content. The method uses two simple rules: wide characters count as half a logical word, and whitespace-split words are clamped to an average length between 3 and 6 characters. Empirical tests show tokens per logical word vary within a ~2× band across tokenizers, compared to a 24× spread for literal word count.", "body_md": "**Logical word count** is a cheap, deterministic, tokenizer-free measure of text length\nthat stays roughly proportional to LLM token counts across natural languages, code, and\nmixed content — while remaining equal to ordinary word count for plain English prose.\n\nIt is defined in two rules:\n\n**Wide (East Asian) characters count as ½ logical word each.** This covers Chinese characters, Japanese kana, Korean Hangul, and CJK punctuation.**Everything else is split on whitespace, and the word count is clamped so the average word length stays between 3 and 6 characters.**Text with very long “words” (compounds, identifiers, URLs, minified code) is counted as`chars / 6`\n\n; text with very short words is counted as`chars / 3`\n\n; typical prose passes through unchanged.\n\nA reference implementation is ~15 lines with no dependencies (see\n[Reference implementation](https://gist.github.com/starred.atom#reference-implementation)). Empirically, tokens per logical\nword stays within a ~2× band for every tokenizer tested — 1.36–2.24 for the GPT-5.6 /\no200k family — across ten languages plus source code and machine formats, versus a 24×\nspread for literal word count and a 3× spread for the common `chars ÷ 4`\n\nrule on the\nsame corpus (see [Validation](https://gist.github.com/starred.atom#validation)).\n\nWord count is the standard unit for measuring text length, but it breaks in two distinct ways: it can under-count content volume, and it can over-count it.\n\nCJK text has few or no word-delimiting spaces; Unicode’s own segmentation standard notes\nthat word boundaries in Chinese, Japanese, or Thai require dictionary lookup. 1 A\n500-character Chinese paragraph counts as a handful of whitespace-delimited “words”\ndespite carrying the information of roughly 250 English words.\nIn our test corpus, an 8,000-character Chinese Wikipedia extract contains just 165\nwhitespace-delimited tokens.\n\nGerman compounds like *Donaudampfschifffahrtsgesellschaft* 2 pack five lexical\nunits into one “word”; agglutinative languages like Finnish average 8+ characters per\nwhitespace-delimited word versus ~5.5 for English.\nIdentical content, materially different word counts.\n\nCode is the hardest case, for two reasons:\n\n**Symbols are rarely space-separated.** The expression `a+b*clamp(x,lo,hi)`\n\nis one\nwhitespace-delimited “word” but contains eight semantic tokens.\nMinified JavaScript averages 44 characters per whitespace-delimited “word” in our\ncorpus; CSV rows and JSON average 10–14.\n\n**Identifiers vary wildly in length.** `x`\n\nand `calculateAverageResponseTime`\n\nare both\none word, but the latter contains five semantic units.\nCode with long descriptive names is under-counted relative to its actual volume.\n\nDocuments that combine prose, code blocks, tables, links, and CJK text have no single word-counting strategy that works across all sections. A README with badges, URLs, and code samples measures 11.7 characters per “word” — double typical prose.\n\nToken counts solve the normalization problem by construction, but they are not stable across model providers, they require a tokenizer vocabulary, and they are orders of magnitude more expensive to compute than a character scan.\n\n**Token counts vary across providers — and across model generations within a provider.**\nThe same 8,000-character samples measured against the latest production tokenizers (July\n2026):[3](https://gist.github.com/starred.atom#user-content-fn-token-variance-310cd646f058931c1f7e1f9c14b67030)\n\n| Sample | GPT-5.6 | Claude Opus 4.8 / Fable 5 | DeepSeek V4 | max deviation |\n|---|---|---|---|---|\n| English Wikipedia | 1,657 | 2,471 | 1,664 | 49% |\n| German Wikipedia | 1,816 | 3,695 | 2,068 | 103% |\n| Chinese Wikipedia | 5,351 | 7,348 | 4,333 | 70% |\n| Japanese Wikipedia | 5,097 | 6,403 | 4,343 | 47% |\n| Korean Wikipedia | 4,529 | 7,916 | 5,026 | 75% |\n| Thai Wikipedia | 3,520 | 7,107 | 3,238 | 119% |\n| Python code | 1,853 | 2,810 | 1,893 | 52% |\n| TypeScript code | 1,907 | 3,518 | 2,004 | 84% |\n\nEven for English the tokenizers diverge by 49%, and for German by 103%. Nor are counts\nstable over time: Anthropic’s newest tokenizer (shared by Claude Opus 4.8, Fable 5, and\nSonnet 5) spends ~37% more tokens on the same English text than the Opus 4.6 tokenizer\ndid, and ~2× on Thai.\nAcademic work documents far larger gaps across languages: the same content can take up\nto 15× more tokens in some languages than in English. 4 A “token count” is\ntherefore not a property of a text — it is a property of a (text, model) pair.\nAny model-independent measure has this cross-tokenizer variance as a floor on its\nachievable accuracy, and that is fine: the goal is a stable measure of\n\n*content volume*that tokens track within a predictable band, not a replacement for a tokenizer when exact counts matter.\n\n**Token counts also require a vocabulary.** Computing them means shipping or calling the\ntarget model’s tokenizer — a dependency that may not exist where length is measured (a\ndatabase check constraint, a CLI, an editor plugin, a non-Python runtime) and that\nchanges when models change.\n\nGiven a text string:\n\n**Wide characters.** Count the characters whose Unicode East Asian Width is Wide or Fullwidth— Han ideographs, Hiragana, Katakana, Hangul syllables, CJK punctuation, and fullwidth forms. Call this[5](https://gist.github.com/starred.atom#user-content-fn-uax11-310cd646f058931c1f7e1f9c14b67030)`wide`\n\n. Each contributes**½** logical word. For splitting purposes, treat each wide character as if it were surrounded by spaces.**Remaining text.** Split what remains on whitespace. Let`words`\n\nbe the number of resulting tokens and`chars`\n\nthe total count of their characters (Unicode code points). Clamp the word count so that average word length stays in`[L, U] = [3, 6]`\n\n:\n\n```\nclamped = clamp(words, chars / 6, chars / 3)\n        = max(chars / 6, min(chars / 3, words))\n\nlogical_words = round(wide / 2 + clamped)\n```\n\nEmpty or whitespace-only text has 0 logical words.\n\nThe clamp activates only when average characters per word falls outside `[3, 6]`\n\n. Most\nEnglish and European prose runs 4.5–5.5 characters per whitespace-delimited word\n(punctuation attached), so for ordinary prose **logical word count equals literal word\ncount** — the measure is backward-compatible where word count already works.\nLong-word text (German and Finnish compounds, identifiers, URLs, minified code, CSV\nrows) is counted as `chars / 6`\n\n; degenerate short-word text (e.g. tables of single\ndigits) is capped at `chars / 3`\n\n.\n\nIn practice the upper bound does nearly all the work; the lower bound is a safety rail that real content rarely hits.\n\n**Canonical definition:** East_Asian_Width ∈ {W, F} (available in Python’s`unicodedata`\n\n, ICU, and most Unicode libraries). Where the property is unavailable, a range-based approximation — the Han and Hangul scripts plus the CJK punctuation-through-Katakana blocks (U+3000–30FF), Katakana phonetic extensions (U+31F0–31FF), and fullwidth forms (U+FF00–FFEF) — produces*identical*counts on all our CJK test samples (kana must be matched by block, not script property: ー U+30FC and ・ U+30FB are Script=Common but Wide). Rare code points (halfwidth Katakana, some symbols) may still differ between the two definitions; the effect on real text is well under 1%.**Korean:** Hangul is space-delimited, but each syllable block encodes 2–3 jamo and tokenizers price Korean like Chinese and Japanese (~1.4 chars/token). Weighting Hangul syllables at ½ overrides the space-based count and empirically aligns Korean with every other language (K = 1.57 vs. 2.54 without the rule; see[Validation](https://gist.github.com/starred.atom#validation)). The localization industry’s GMX-V standard likewise groups Korean with the logographic scripts.[6](https://gist.github.com/starred.atom#user-content-fn-gmxv-310cd646f058931c1f7e1f9c14b67030)**Emoji** are East_Asian_Width Wide, so they count ½ each. Tokenizers typically spend 1–3 tokens per emoji; at realistic emoji densities the error is negligible.\n\nIndependent conventions, developed for different purposes, converge on roughly two CJK characters per English-word-equivalent:\n\n| Convention | Effective rate | Purpose |\n|---|---|---|\n| Translation-industry rule of thumb | ~1,000 Japanese chars ≈ 400–500 English words; 1,000 Chinese chars ≈ 650–750 English words\n|\nQuoting/billing |\n| Japanese 原稿用紙 (genkō yōshi) manuscript page | 400 chars ≈ 200–250 English words\n|\nPublishing |\n| Twitter/X character limit | CJK weighted 2× (280-unit limit = 140 CJK chars)\n|\nLength limiting |\n| Medium & Hugo reading speed | 500 CJK chars/min vs. ~212–265 words/min ≈ 2–2.4 chars per word\n|\nReading time |\n| ICU dictionary segmentation | Chinese words average ~1.5–2 chars\n|\nLinguistic segmentation |\n| Modern LLM tokenizers (measured) | one CJK char ≈ 0.7 tokens ≈ half the ~1.35 tokens of an English word | Token accounting |\n\nThe tokenizer alignment is the one this spec is calibrated to: weighting CJK characters at ½ word puts Chinese, Japanese, and Korean at 1.54–1.57 tokens per logical word under o200k — statistically indistinguishable from French (1.50) and German (1.57).\n\nGMX-V, the localization industry’s published volume standard, uses somewhat higher\ndivisors for *source-word* counts (Chinese ÷ 2.8, Japanese ÷ 3.0, Korean ÷ 3.3) 6 —\nit estimates the number of words a native segmentation would find, whereas logical word\ncount (like the billing ratios above) targets volume equivalence.\nSimple word-counting tools mostly land on 1× instead: Microsoft Word and memoQ count\neach CJK character as one word,\n\nand Unicode’s default word segmentation breaks between every pair of ideographs.\n\n[12](https://gist.github.com/starred.atom#user-content-fn-word-count-tools-310cd646f058931c1f7e1f9c14b67030)Half a word per character sits deliberately between “one word per character” (too high: overweights CJK ~2× in any cross-language comparison) and GMX-V’s ~⅓ (too low for token tracking).\n\n[1](https://gist.github.com/starred.atom#user-content-fn-uax29-310cd646f058931c1f7e1f9c14b67030)BPE tokenizers merge frequent byte sequences until each token carries roughly one\n“meaning-sized” chunk. 13 Measured on large samples, modern large-vocabulary\ntokenizers (o200k, cl100k, Llama 3, Gemma 2) all land near\n\n**4.2–4.3 characters per token for English prose** and 4.0–4.8 for source code;\n\nvendor rules of thumb agree (OpenAI: ~4 chars or ¾ of a word per token; Anthropic: ~3.5 chars; Google: ~4 chars).\n\n[14](https://gist.github.com/starred.atom#user-content-fn-chars-per-token-310cd646f058931c1f7e1f9c14b67030)(Anthropic’s newest tokenizer is denser — ~2.7 characters per token on our English sample — but uniformly so across content types, which shifts K by a constant factor without widening the band the clamp is designed to hold.)\n\n[15](https://gist.github.com/starred.atom#user-content-fn-vendor-rules-310cd646f058931c1f7e1f9c14b67030)The bounds bracket that range without a vocabulary table:\n\n**U = 6** is the point where long-word content (compounds, identifiers, URLs) stops gaining count from packing more characters into each word.`chars / 6`\n\ntracks the ~4–5 chars/token that tokenizers actually spend on such content, at a stable ratio of ~1.3–1.5 tokens per logical word.**L = 3** caps degenerate short-word content. Real prose almost never averages below 3 (even Korean, with dense syllable blocks, averages 3.5); the bound exists so that pathological inputs (single-character tables, heavily spaced tokens) cannot inflate the count more than ~1.5× over prose.- Everything between 3 and 6 chars/word passes through, preserving\n`logical_words = literal words`\n\nfor ordinary prose in English, French, Spanish, Russian, Hebrew, and most other spaced languages.\n\nTightening the bounds (e.g. `[3, 5]`\n\n) narrows the token-tracking band another ~10% but\nsacrifices exact backward compatibility with English word count; loosening them (e.g.\n`[2, 7]`\n\n) admits outliers without benefit.\n`[3, 6]`\n\nwith weight ½ is the recommended convention; all three constants are parameters\nin the reference implementation for applications that want to recalibrate.\n\nPython (canonical):\n\n``` python\nimport unicodedata\n\ndef logical_word_count(text: str, lower: float = 3.0, upper: float = 6.0,\n                       wide_weight: float = 0.5) -> int:\n    wide = 0\n    rest = []\n    for c in text:\n        if unicodedata.east_asian_width(c) in (\"W\", \"F\"):\n            wide += 1\n            rest.append(\" \")\n        else:\n            rest.append(c)\n    words = \"\".join(rest).split()\n    chars = sum(len(w) for w in words)\n    clamped = max(chars / upper, min(chars / lower, len(words)))\n    return round(wide * wide_weight + clamped)\n```\n\nTypeScript (script-range approximation of East_Asian_Width; identical results on real-world text):\n\n``` js\nconst WIDE =\n  /[\\p{Script=Han}\\p{Script=Hangul}\\u3000-\\u30FF\\u31F0-\\u31FF\\uFF00-\\uFFEF]/u;\n\nfunction logicalWordCount(\n  text: string, lower = 3, upper = 6, wideWeight = 0.5,\n): number {\n  let wide = 0;\n  const rest: string[] = [];\n  for (const ch of text) { // iterates code points, not UTF-16 units\n    if (WIDE.test(ch)) {\n      wide++;\n      rest.push(\" \");\n    } else {\n      rest.push(ch);\n    }\n  }\n  const words = rest.join(\"\").split(/\\s+/).filter(Boolean);\n  const chars = words.reduce((n, w) => n + [...w].length, 0);\n  const clamped = Math.max(chars / upper, Math.min(chars / lower, words.length));\n  return Math.round(wide * wideWeight + clamped);\n}\n```\n\nBoth are O(n), allocation-light, and dependency-free.\n\nMeasured with the companion script (`validate_lwc.py`\n\n) on 8,000-character samples:\nWikipedia extracts in ten languages (Hebrew and Thai are reported under\n[Limitations](https://gist.github.com/starred.atom#limitations)), real source files, minified JS, CSV, JSON, and a\nmixed-content README. Token counts are GPT-5.6 via the OpenAI API (the o200k tokenizer\nfamily); K = tokens / logical words.\n\n| Sample | chars | literal words | logical words | tokens | K |\n|---|---|---|---|---|---|\n| English Wikipedia | 6,775 | 1,219 | 1,219 | 1,657 | 1.36 |\n| French Wikipedia | 6,756 | 1,222 | 1,222 | 1,838 | 1.50 |\n| German Wikipedia | 6,921 | 1,063 | 1,154 | 1,816 | 1.57 |\n| Russian Wikipedia | 6,998 | 995 | 1,166 | 1,939 | 1.66 |\n| Finnish Wikipedia | 7,104 | 868 | 1,184 | 2,309 | 1.95 |\n| Chinese Wikipedia | 7,807 | 165 | 3,400 | 5,351 | 1.57 |\n| Japanese Wikipedia | 7,642 | 300 | 3,300 | 5,097 | 1.54 |\n| Korean Wikipedia | 6,181 | 1,781 | 2,888 | 4,529 | 1.57 |\n| Python code | 5,709 | 862 | 952 | 1,853 | 1.95 |\n| TypeScript code | 6,800 | 865 | 1,133 | 1,907 | 1.68 |\n| Markdown README | 7,314 | 627 | 1,219 | 2,178 | 1.79 |\n| Minified JavaScript | 7,825 | 176 | 1,304 | 2,889 | 2.22 |\n| CSV data | 7,462 | 539 | 1,244 | 2,786 | 2.24 |\n| JSON | 7,037 | 560 | 1,173 | 2,168 | 1.85 |\n\nProse, code, and CJK all land in **K = 1.36–1.95** (a 1.43× band).\nPunctuation-dense machine formats run hotter, up to ~2.2. Compare the alternatives on\nthe identical corpus:\n\n| Measure | tokens ÷ measure, across samples | spread |\n|---|---|---|\nlogical words |\n1.36 – 2.24 | 1.65× |\n| literal words | 1.36 – 32.4 | 24× |\n| chars ÷ 4 | 0.98 – 2.9 | 3.0× |\n\nThe same holds for the other tokenizer families. On the identical corpus (including the Hebrew and Thai limitation cases), tokens per logical word spans a 2.0× band for GPT-5.6, 2.1× for DeepSeek V4, and 2.9× for the Claude Opus 4.8 / Fable 5 tokenizer — each centered on its own K — while literal word count spans 19–24× on every one of them.\n\nThe remaining variation is largely irreducible: the tokenizers themselves disagree by\n37–119% on identical text (see the table in\n[Motivation](https://gist.github.com/starred.atom#why-not-just-use-token-counts)), so no model-free measure can track any\nsingle model much more tightly than models track each other.\n\nFor a quick estimate against o200k-family tokenizers (GPT-5.x; DeepSeek V4 runs ~10% higher):\n\n```\nestimated_tokens ≈ logical_words × 1.6\n```\n\nThis lands within roughly ±25% across prose (any script), code, and markdown. Useful refinements:\n\n**Clean English prose:** use ×1.35 — which is the familiar “1 word ≈ 4⁄3 tokens” vendor rule,recovered exactly because logical words equal literal words for such text.[15](https://gist.github.com/starred.atom#user-content-fn-vendor-rules-310cd646f058931c1f7e1f9c14b67030)**Punctuation-dense machine formats**(minified code, CSV, logs): use ×2.2, or accept under-estimation.** Model-specific correction:**apply a per-tokenizer factor — DeepSeek V4 runs ~1.1× o200k, and Anthropic’s newest tokenizer (Claude Opus 4.8 / Fable 5 / Sonnet 5) runs ~1.5–2× o200k depending on content, i.e. tokens ≈ logical_words × 2.75 overall. This correction is roughly constant per model, unlike the correction needed for raw word count, which varies by an order of magnitude with text type.\n\nWhen exact counts matter (billing, hard context limits), use the model’s tokenizer; logical word count is for the much larger class of uses where a stable, cheap, model-free number is worth ±25%.\n\n**Cheap:** O(n) single pass, no vocabulary, no dependencies.**Deterministic and model-free:** same input, same count, forever.**Backward-compatible:** equals literal word count for ordinary English (and most European-language) prose.**Script-aware but language-blind:** uses only character properties; no language detection, dictionaries, or segmentation models.**Monotonic:** appending non-whitespace content never decreases the count.**Additive:** splitting text at whitespace boundaries and summing the parts’ counts approximates the whole (within rounding), so counts can be computed incrementally or in parallel.**Bounded token ratio:** every tokenizer tested spends a near-constant number of tokens per logical word — within a ~2–3× band across all content types, versus ~20–24× for literal word count.\n\nStated plainly, since a measure is only trustworthy if its failure modes are known:\n\n**Unspaced non-CJK scripts**(Thai, Lao, Khmer, Myanmar) are not covered by the wide-character rule; they fall to the clamp’s`chars / 6`\n\nfloor. That happens to match GMX-V’s convention for Thai (chars ÷ 6),but o200k still spends ~2.75 tokens per logical word on Thai (Anthropic’s newest tokenizer, ~5.6) — high fertility for these scripts is a property of the tokenizers, not of the measure. Applications centered on these scripts should add a per-script weight (Thai ≈ ¼–⅓ word/char) by the same method used here for CJK.[6](https://gist.github.com/starred.atom#user-content-fn-gmxv-310cd646f058931c1f7e1f9c14b67030)**Non-Latin alphabetic scripts**(Hebrew, Greek, Devanagari, …) count correctly as*volume*— their word counts are true word counts — but tokenizers charge them 1.9–2.3 tokens per logical word versus 1.35 for English (o200k; other tokenizer families show the same skew at their own scale). This is the well-documented tokenizer-fairness gap,[4](https://gist.github.com/starred.atom#user-content-fn-petrov-310cd646f058931c1f7e1f9c14b67030)and logical word count deliberately does not bend the volume measure to compensate for it.[16](https://gist.github.com/starred.atom#user-content-fn-ahia-310cd646f058931c1f7e1f9c14b67030)**Machine formats**(minified code, CSV, dense JSON) run ~1.4× hotter in tokens than prose. The clamp keeps them bounded (literal word count is off by 24× here), but a single multiplier will under-estimate their token cost.**It is not a linguistic word count.** For Chinese, ½ word per character approximates dictionary segmentation (Chinese words average 1.5–2 characters), but no claim is made that`logical_words`\n\nmatches what a human or a segmenter would count as words in any specific language. It is a normalized volume unit, like a “standard page” — not a lexical analysis.\n\nNo existing term or formula was found for this specific construction (a chars-per-word\nclamp plus a wide-character weight), and the name “logical word count” appears to be\nunclaimed. (\"Standard word\" already means 5 keystrokes in typing measurement; 17\n“weighted word count” means fuzzy-match discounts in CAT tools.)\nThe pieces, however, have deep roots:\n\n**GMX-V (LISA → ETSI GS LIS 004)**— the localization industry’s word/character count standard — is the closest precedent. Version 1.0 (2007) refused to segment unspaced scripts and required character counts; version 2.0 (2012) added normative per-script divisors to convert characters to words: Chinese ÷ 2.8, Japanese ÷ 3.0, Korean ÷ 3.3, Thai ÷ 6, described as “acknowledged best practice within the Localization Industry.”Logical word count generalizes the same move — chars ÷ factor — and extends it to code and mixed content via the clamp.[6](https://gist.github.com/starred.atom#user-content-fn-gmxv-310cd646f058931c1f7e1f9c14b67030)**CJK character-as-word conventions:** Microsoft Word’s “Words” statistic counts each East Asian character as a word (with a separate Asian-characters statistic); memoQ computes source words as Asian characters + non-Asian words; WordPress locale packs switch Japanese and Chinese to character counting outright.[12](https://gist.github.com/starred.atom#user-content-fn-word-count-tools-310cd646f058931c1f7e1f9c14b67030)**Weighted length limits:** Twitter/X’s 280-unit limit weights code points beyond a small Latin-ish set (roughly through U+10FF plus general punctuation) at 2, so CJK posts max out at 140 characters — a shipped, planet-scale precedent for weighting wide characters at exactly 2× Latin.[9](https://gist.github.com/starred.atom#user-content-fn-twitter-310cd646f058931c1f7e1f9c14b67030)**Reading-time estimators:** Medium uses ~265 words/min for spaced languages and 500 characters/min for CJK; Hugo and the`reading-time`\n\nnpm package (used by Gatsby, Ghost analogues) count CJK per-character against a ~2.4× higher rate.[10](https://gist.github.com/starred.atom#user-content-fn-medium-310cd646f058931c1f7e1f9c14b67030)**Character-based billing units:** Germany’s court-interpreting law (§ 11 JVEG) prices translation per 55 keystrokes (\"je angefangene 55 Anschläge\"); publishing standard pages (Normseite: 1,500–1,800 chars) and typing’s 5-character “standard word” all define length by characters when word counts are unreliable or unfair.[18](https://gist.github.com/starred.atom#user-content-fn-jveg-310cd646f058931c1f7e1f9c14b67030)Amazon’s KENPC normalizes e-book length to a synthetic page unit for paying authors.[17](https://gist.github.com/starred.atom#user-content-fn-wpm-310cd646f058931c1f7e1f9c14b67030)[19](https://gist.github.com/starred.atom#user-content-fn-kenpc-310cd646f058931c1f7e1f9c14b67030)**Character length as a text statistic:** the Automated Readability Index (1967) used chars/word as a machine-friendly complexity proxy precisely because characters were cheap to count.[20](https://gist.github.com/starred.atom#user-content-fn-ari-310cd646f058931c1f7e1f9c14b67030)**Tokenizer research:** subword*fertility*(tokens per word) is the standard metric for tokenizer efficiency;the unfairness of token counts across languages is documented up to 15×.[21](https://gist.github.com/starred.atom#user-content-fn-fertility-310cd646f058931c1f7e1f9c14b67030)[4](https://gist.github.com/starred.atom#user-content-fn-petrov-310cd646f058931c1f7e1f9c14b67030)BPE itself[16](https://gist.github.com/starred.atom#user-content-fn-ahia-310cd646f058931c1f7e1f9c14b67030)solves normalization by construction, at the cost of a vocabulary and model dependence.[13](https://gist.github.com/starred.atom#user-content-fn-sennrich2016-310cd646f058931c1f7e1f9c14b67030)**Tokenizer-free token estimators**(the`tokenx`\n\nnpm package; LangChain’s`count_tokens_approximately`\n\n; Semantic Kernel’s chars ÷ 4) estimate tokens for a*specific*model family from character ratios.Logical word count differs in intent: it is a stable text measure first — human-legible, word-count compatible — that happens to track tokens within a known band, rather than a model-specific predictor.[22](https://gist.github.com/starred.atom#user-content-fn-estimators-310cd646f058931c1f7e1f9c14b67030)\n\nAll measurements in this document come from the companion script `validate_lwc.py`\n\n(published alongside this document), which fetches the public samples, computes literal\nwords, logical words, and exact token counts via provider APIs, and prints the\ncomparison tables:\n\n```\nuv run validate_lwc.py                  # o200k locally; other providers with API keys\nuv run validate_lwc.py --lower 3 --upper 6 --wide-weight 0.5\nuv run validate_lwc.py --list-samples\n```\n\nWith no API keys, OpenAI counts fall back to local tiktoken (o200k_base — measured\nidentical to the API counts modulo ~6 tokens of message framing).\nSet `OPENAI_API_KEY`\n\n, `ANTHROPIC_API_KEY`\n\n, and/or `DEEPSEEK_API_KEY`\n\nto reproduce the\ncross-model table with exact provider counts; Gemini and xAI models are supported via\n`--models`\n\nwith their keys.\nThe script’s dependencies are pinned under a two-week cooling-off policy (uv’s\n`exclude-newer`\n\n).\n\n## Footnotes\n\n-\nUnicode Standard Annex #29, “Unicode Text Segmentation,” §4 Word Boundaries: default rules break between every pair of ideographs (rule WB999), and “reliable detection of word boundaries in languages such as Thai, Lao, Chinese, or Japanese requires the use of dictionary lookup or other mechanisms.”\n\n[https://unicode.org/reports/tr29/](https://unicode.org/reports/tr29/)[↩](https://gist.github.com/starred.atom#user-content-fnref-uax29-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-uax29-2-310cd646f058931c1f7e1f9c14b67030)2 -\nDanube–steam–ship–journey–company; the Erste Donau-Dampfschiffahrts-Gesellschaft (founded 1829) is the canonical example of German nominal compounding.\n\n[https://en.wikipedia.org/wiki/Donaudampfschiffahrtsgesellschaft](https://en.wikipedia.org/wiki/Donaudampfschiffahrtsgesellschaft)[↩](https://gist.github.com/starred.atom#user-content-fnref-compound-310cd646f058931c1f7e1f9c14b67030) -\nMeasured 2026-07-12 with\n\n`validate_lwc.py`\n\non 8,000-character truncations of Wikipedia articles and open-source code. GPT-5.6 = OpenAI chat completions API`usage.prompt_tokens`\n\n(the three GPT-5.6 variants — luna, sol, terra — return identical counts, matching tiktoken’s o200k_base within ~6 tokens of message framing); Claude = Anthropic`count_tokens`\n\nAPI (Claude Opus 4.8, Fable 5, and Sonnet 5 return identical counts — one shared tokenizer); DeepSeek = OpenAI-compatible completions API (deepseek-v4-pro and v4-flash identical). An earlier five-provider run that included Gemini 3.1 Pro and Grok-4 showed the same picture (35–79% spread).[↩](https://gist.github.com/starred.atom#user-content-fnref-token-variance-310cd646f058931c1f7e1f9c14b67030) -\nPetrov, La Malfa, Torr, and Bibi, “Language Model Tokenizers Introduce Unfairness Between Languages,” NeurIPS 2023. arXiv:2305.15425. “The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases” (Shan vs. English under cl100k; even the cheapest languages carry a ~50% premium over English).\n\n[https://arxiv.org/abs/2305.15425](https://arxiv.org/abs/2305.15425)[↩](https://gist.github.com/starred.atom#user-content-fnref-petrov-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-petrov-2-310cd646f058931c1f7e1f9c14b67030)2[↩](https://gist.github.com/starred.atom#user-content-fnref-petrov-3-310cd646f058931c1f7e1f9c14b67030)3 -\nUnicode Standard Annex #11, “East Asian Width.”\n\n[https://www.unicode.org/reports/tr11/](https://www.unicode.org/reports/tr11/)[↩](https://gist.github.com/starred.atom#user-content-fnref-uax11-310cd646f058931c1f7e1f9c14b67030) -\nETSI GS LIS 004 V2.0.0 (2012-07), “GMX-V: Global Information Management Metrics eXchange — Volume” (successor to the LISA standard), clause 4.2.13 “Logographic Scripts”: word counts for unspaced scripts are derived from character counts via fixed factors — Chinese 2.8, Japanese 3.0, Korean 3.3, Thai 6.0 — “based on acknowledged best practice within the Localization Industry.”\n\n[https://www.etsi.org/deliver/etsi_gs/lis/001_099/004/02.00.00_60/gs_lis004v020000p.pdf](https://www.etsi.org/deliver/etsi_gs/lis/001_099/004/02.00.00_60/gs_lis004v020000p.pdf)[↩](https://gist.github.com/starred.atom#user-content-fnref-gmxv-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-gmxv-2-310cd646f058931c1f7e1f9c14b67030)2[↩](https://gist.github.com/starred.atom#user-content-fnref-gmxv-3-310cd646f058931c1f7e1f9c14b67030)3[↩](https://gist.github.com/starred.atom#user-content-fnref-gmxv-4-310cd646f058931c1f7e1f9c14b67030)4 -\nE.g. AnyCount, “Word count in Oriental languages”: 1,000 Chinese characters ≈ 650–750 English words (\n\n[https://www.anycount.com/word-count-worldwide/word-count-in-oriental-languages/](https://www.anycount.com/word-count-worldwide/word-count-in-oriental-languages/)); 1StopAsia gives the same Chinese figures and recommends character counts for Chinese/Japanese/Thai ([https://www.1stopasia.com/blog/word-count-vs-character-count-for-asian-languages/](https://www.1stopasia.com/blog/word-count-vs-character-count-for-asian-languages/)).[↩](https://gist.github.com/starred.atom#user-content-fnref-cjk-ratios-310cd646f058931c1f7e1f9c14b67030) -\nSWET (Society of Writers, Editors and Translators, Tokyo), “What’s in a Page?”: the 400-character genkō yōshi manuscript page corresponds to roughly 200–250 English words.\n\n[https://swet.jp/articles/article/whats_in_a_page/_C31](https://swet.jp/articles/article/whats_in_a_page/_C31)[↩](https://gist.github.com/starred.atom#user-content-fnref-swet-310cd646f058931c1f7e1f9c14b67030) -\nX developer documentation, “Counting characters,” and the open-source twitter-text library config (v3.json): code points U+0000–10FF plus selected general-punctuation ranges weigh 1; everything else (all CJK, Hangul, emoji) weighs 2; URLs count as 23.\n\n[https://docs.x.com/fundamentals/counting-characters](https://docs.x.com/fundamentals/counting-characters),[https://github.com/twitter/twitter-text/blob/master/config/v3.json](https://github.com/twitter/twitter-text/blob/master/config/v3.json)[↩](https://gist.github.com/starred.atom#user-content-fnref-twitter-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-twitter-2-310cd646f058931c1f7e1f9c14b67030)2 -\nMedium Help Center, “Read time”: “roughly 265 WPM… For posts in Chinese, Japanese and Korean, it’s a function of number of characters (500 characters/min).” Hugo (\n\n`hugolib/page__content.go`\n\n) uses the same 500 chars/min for CJK vs. ~213 wpm otherwise, and counts runes per CJK token when`hasCJKLanguage`\n\nis set. The`reading-time`\n\nnpm package counts each CJK character as one word.[https://help.medium.com/hc/en-us/articles/214991667](https://help.medium.com/hc/en-us/articles/214991667),[https://github.com/gohugoio/hugo](https://github.com/gohugoio/hugo),[https://github.com/ngryman/reading-time](https://github.com/ngryman/reading-time)[↩](https://gist.github.com/starred.atom#user-content-fnref-medium-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-medium-2-310cd646f058931c1f7e1f9c14b67030)2 -\nICU segments Chinese and Japanese with a frequency dictionary (cjdict) rather than Unicode default rules; dictionary-segmented Chinese words average roughly 1.5–2 characters.\n\n[https://unicode-org.github.io/icu/userguide/boundaryanalysis/](https://unicode-org.github.io/icu/userguide/boundaryanalysis/)[↩](https://gist.github.com/starred.atom#user-content-fnref-icu-310cd646f058931c1f7e1f9c14b67030) -\nMicrosoft Word: “In East Asian languages, each character counts as a word” (NumChars documentation), with a separate\n\n`wdStatisticFarEastCharacters`\n\nstatistic ([https://learn.microsoft.com/en-us/office/vba/api/word.wdstatistic](https://learn.microsoft.com/en-us/office/vba/api/word.wdstatistic)). memoQ: “Source words = Asian characters + non-Asian words” for Chinese/Japanese sources ([https://docs.memoq.com/current/en/Workspace/statistics.html](https://docs.memoq.com/current/en/Workspace/statistics.html)). WordPress: locale packs translate the word-count type to character counting for Japanese and Chinese (`WP_Locale::get_word_count_type()`\n\n).[↩](https://gist.github.com/starred.atom#user-content-fnref-word-count-tools-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-word-count-tools-2-310cd646f058931c1f7e1f9c14b67030)2 -\nSennrich, Haddow, and Birch, “Neural Machine Translation of Rare Words with Subword Units,” ACL 2016 — the canonical BPE-for-NLP paper.\n\n[https://aclanthology.org/P16-1162/](https://aclanthology.org/P16-1162/)[↩](https://gist.github.com/starred.atom#user-content-fnref-sennrich2016-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-sennrich2016-2-310cd646f058931c1f7e1f9c14b67030)2 -\nMeasured on ~300k characters of English prose (Pride and Prejudice) and ~165k characters of Python stdlib source: cl100k_base 4.24/4.81 chars per token (prose/code), o200k_base 4.28/4.77, Llama 3 4.25/4.81, Gemma 2 4.29/4.03. Shorter, punctuation- or link-heavy samples run lower (3.1–3.6 in the validation corpus).\n\n[↩](https://gist.github.com/starred.atom#user-content-fnref-chars-per-token-310cd646f058931c1f7e1f9c14b67030) -\nOpenAI: “1 token ≈ 4 characters… ≈ ¾ of a word; 100 tokens ≈ 75 words” (\n\n[https://help.openai.com/en/articles/4936856](https://help.openai.com/en/articles/4936856)) and “1 token is approximately 4 characters or 0.75 words for English text” ([https://developers.openai.com/api/docs/concepts](https://developers.openai.com/api/docs/concepts)). Anthropic: “a token approximately represents 3.5 English characters” ([https://platform.claude.com/docs/en/docs/resources/glossary](https://platform.claude.com/docs/en/docs/resources/glossary)). Google: “a token is equivalent to about 4 characters… 100 tokens is equal to about 60-80 English words” ([https://ai.google.dev/gemini-api/docs/tokens](https://ai.google.dev/gemini-api/docs/tokens)).[↩](https://gist.github.com/starred.atom#user-content-fnref-vendor-rules-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-vendor-rules-2-310cd646f058931c1f7e1f9c14b67030)2 -\nAhia et al., “Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models,” EMNLP 2023. arXiv:2305.13707. Documents ~5× token-count and API-cost premiums for non-Latin-script languages.\n\n[https://aclanthology.org/2023.emnlp-main.614/](https://aclanthology.org/2023.emnlp-main.614/)[↩](https://gist.github.com/starred.atom#user-content-fnref-ahia-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-ahia-2-310cd646f058931c1f7e1f9c14b67030)2 -\nWords-per-minute measurement standardizes a “word” as 5 characters or keystrokes (including spaces and punctuation).\n\n[https://en.wikipedia.org/wiki/Words_per_minute](https://en.wikipedia.org/wiki/Words_per_minute)[↩](https://gist.github.com/starred.atom#user-content-fnref-wpm-310cd646f058931c1f7e1f9c14b67030)[↩](https://gist.github.com/starred.atom#user-content-fnref-wpm-2-310cd646f058931c1f7e1f9c14b67030)2 -\n§ 11 JVEG (Justizvergütungs- und -entschädigungsgesetz): translation honoraria “für jeweils angefangene 55 Anschläge des schriftlichen Textes” (per commenced 55 keystrokes).\n\n[https://www.gesetze-im-internet.de/jveg/__11.html](https://www.gesetze-im-internet.de/jveg/__11.html). Normseite: 30 lines × 60 keystrokes = 1,800 characters (publishing convention); VG Wort uses 1,500.[https://de.wikipedia.org/wiki/Normseite](https://de.wikipedia.org/wiki/Normseite)[↩](https://gist.github.com/starred.atom#user-content-fnref-jveg-310cd646f058931c1f7e1f9c14b67030) -\nAmazon KDP, “Royalties in Kindle Unlimited”: KENPC (Kindle Edition Normalized Page Count) normalizes page count using standard formatting settings; the formula is unpublished (author analyses estimate ~170–200 words per page).\n\n[https://kdp.amazon.com/en_US/help/topic/G201541130](https://kdp.amazon.com/en_US/help/topic/G201541130)[↩](https://gist.github.com/starred.atom#user-content-fnref-kenpc-310cd646f058931c1f7e1f9c14b67030) -\nSenter and Smith, “Automated Readability Index,” AMRL-TR-66-220, 1967. Formula: ARI = 4.71 × (chars/words) + 0.5 × (words/sentences) − 21.43.\n\n[https://apps.dtic.mil/sti/tr/pdf/AD0667273.pdf](https://apps.dtic.mil/sti/tr/pdf/AD0667273.pdf)[↩](https://gist.github.com/starred.atom#user-content-fnref-ari-310cd646f058931c1f7e1f9c14b67030) -\nÁcs, “Exploring BERT’s Vocabulary” (2019) defined subword fertility — average subwords per word — borrowing the term from statistical MT (Brown et al. 1993); Rust et al., “How Good is Your Tokenizer?” (ACL 2021) established it as the standard tokenizer-efficiency metric.\n\n[https://juditacs.github.io/2019/02/19/bert-tokenization-stats.html](https://juditacs.github.io/2019/02/19/bert-tokenization-stats.html),[https://aclanthology.org/2021.acl-long.243/](https://aclanthology.org/2021.acl-long.243/)[↩](https://gist.github.com/starred.atom#user-content-fnref-fertility-310cd646f058931c1f7e1f9c14b67030) -\ntokenx: character-ratio estimation with per-language overrides, “~96% accuracy” (\n\n[https://github.com/johannschopplich/tokenx](https://github.com/johannschopplich/tokenx)). LangChain`count_tokens_approximately`\n\n: chars ÷ 4 plus per-message overhead. Microsoft Semantic Kernel TextChunker: length ÷ 4.[↩](https://gist.github.com/starred.atom#user-content-fnref-estimators-310cd646f058931c1f7e1f9c14b67030)", "url": "https://wpnews.pro/news/logical-word-count-a-cheap-tokenizer-free-measure-of-text-length-that-tracks-llm", "canonical_source": "https://gist.github.com/jlevy/0d6d87885f6d85f31440e58b8cfce663", "published_at": "2026-07-12 22:47:23+00:00", "updated_at": "2026-07-12 23:09:34.973210+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "developer-tools"], "entities": ["GPT-5.6", "Claude Opus 4.8", "DeepSeek V4"], "alternates": {"html": "https://wpnews.pro/news/logical-word-count-a-cheap-tokenizer-free-measure-of-text-length-that-tracks-llm", "markdown": "https://wpnews.pro/news/logical-word-count-a-cheap-tokenizer-free-measure-of-text-length-that-tracks-llm.md", "text": "https://wpnews.pro/news/logical-word-count-a-cheap-tokenizer-free-measure-of-text-length-that-tracks-llm.txt", "jsonld": "https://wpnews.pro/news/logical-word-count-a-cheap-tokenizer-free-measure-of-text-length-that-tracks-llm.jsonld"}}