{"slug": "ctok-reconstructed-claude-tokenizer", "title": "Ctok: Reconstructed Claude Tokenizer", "summary": "Ctok, an unofficial open-source library, reconstructs Anthropic's Claude tokenizer offline, reporting exact token counts for 1,664,940 v3 and 1,722,961 v4.7 texts with zero under-counts. The library supports Claude model generations from v3 (Claude 3 through Opus 4.6) to v5 (Opus 5 and Sonnet 5), with version strings compared component-wise, and is not affiliated with Anthropic.", "body_md": "`ctok`\n\nreconstructs Claude token counts offline, with no API call, network access, or runtime\ndependencies. It is unofficial and is not affiliated with Anthropic.\n\nThe reconstruction targets counts. Claude does not expose token boundaries, so `tokenize()`\n\nreturns\none valid minimum-cost tiling, not a claim about Anthropic's exact segmentation. The research behind\nthe model is described in [On the biology of Claude's tokenizer](https://tokencontributions.substack.com/p/on-the-biology-of-claudes-tokenizer).\n\n``` python\nfrom ctok import token_count, tokenize\n\ntoken_count(\"hello, world\")           # 10, using the v3 family\ntoken_count(\"hello, world\", \"4.7\")    # 15\ntoken_count(\"hello, world\", \"5.0\")    # 10\n\ntokens = tokenize(\"NASA likes tokenizers\")\nassert len(tokens) == token_count(\"NASA likes tokenizers\")\n```\n\nThe command-line interface prints the marked stream and its tiling:\n\n```\nctok \"hello, world\"\n```\n\n`version`\n\nis always a string, e.g. `\"4.7\"`\n\n, compared component by component — `\"4.10\"`\n\nsorts after\n`\"4.9\"`\n\n, not below `\"4.2\"`\n\n. A `float`\n\ncan't make that distinction (Python collapses the literal\n`4.10`\n\nto `4.1`\n\nbefore any code here sees it), so a non-`str`\n\nversion raises `TypeError`\n\n.\n\n| requested version | family | model generation |\n|---|---|---|\n`\"3.0\" <= version < \"4.7\"` |\nv3, the default | Claude 3 through Opus 4.6 |\n`\"4.7\" <= version < \"5.0\"` |\nv4.7 | Opus 4.7 through 4.9 |\n`version >= \"5.0\"` |\nv5 | Opus 5 and Sonnet 5 |\n\nv5 is v4.7 with a slightly different fixed overhead.\n\nFor one user message, `ctok`\n\n:\n\n- normalizes the text, including NFC and family-specific quote folding;\n- rewrites it into a stream with word, case, and byte markers;\n- finds a minimum-cost tiling over the measured vocabulary and UTF-8 byte fallback;\n- adds the measured message frame.\n\n`token_count(text)`\n\nis `len(tokenize(text))`\n\n. The output notation makes internal structure visible:\n\n| notation | meaning |\n|---|---|\n`⟨bow⟩` , `⟨eow⟩` |\nword boundaries |\n`⟨shift⟩` , `⟨caps⟩` |\ncase rewrites |\n`⟨0xNN⟩` |\na byte-fallback token |\n`⟨pad⟩` |\npart of the single-message frame |\n\nThese results compare `ctok`\n\nwith recorded `count_tokens`\n\nresponses:\n\n| corpus | role | v3 exact | v4.7 exact |\n|---|---|---|---|\n| Goldfish, 350 languages and 350,000 rows | mining | 350,000 | 350,000 |\n| MultiPL-E, 22 programming languages | held out | 22 | 22 |\n| Rosetta Code, 1,741 documents | mining | 1,741 | 1,741 |\n| Rosetta Code, separate 250 documents | mining | 250 | 250 |\n| UDHR, 501 languages | mining (in-sample since 2026-08-12) | 501 | 501 |\n\nv5 has the same content result as v4.7 because it uses the same vocabulary.\n\nThe stored measurement sets contain no under-counts: 0 of 1,664,940 v3 texts and 0 of 1,722,961 v4.7 texts. This is an empirical result, not a guarantee for arbitrary input. Goldfish, Rosetta and (since its final six pieces were selected by bisecting it) UDHR may select candidates; MultiPL-E never does, and is the one remaining held-out corpus in this table.\n\nRun the public gates with:\n\n```\nuv run pytest\nuv run python tests/gates.py --markdown\n```\n\nThe two vocabulary files contain 48,645 v3 pieces and 15,240 v4.7 pieces. Every entry has a fixed membership witness or is one of the structural marker atoms checked by the test suite.\n\n``` python\nfrom ctok import pieces, witness\n\nlen(pieces(\"4.7\"))                       # 15240\nwitness(\"⟨bow⟩the⟨eow⟩\", \"4.7\")\n# {'probe': 'the', 'raw': 12, 'kind': 'raw'}\n```\n\nA witness says that one marked piece costs one token in a calibrated probe. It does not prove the\nencoder rewrite or resolve ties between equal-cost tilings. `tests/test_witness.py`\n\nchecks every\npublished witness and requires complete witnessed-or-special coverage.\n\nMIT", "url": "https://wpnews.pro/news/ctok-reconstructed-claude-tokenizer", "canonical_source": "https://github.com/sanderland/ctok", "published_at": "2026-08-15 20:10:23+00:00", "updated_at": "2026-08-15 20:41:08.132328+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools"], "entities": ["Ctok", "Anthropic", "Claude", "Opus 4.6", "Opus 4.7", "Opus 5", "Sonnet 5", "MultiPL-E"], "alternates": {"html": "https://wpnews.pro/news/ctok-reconstructed-claude-tokenizer", "markdown": "https://wpnews.pro/news/ctok-reconstructed-claude-tokenizer.md", "text": "https://wpnews.pro/news/ctok-reconstructed-claude-tokenizer.txt", "jsonld": "https://wpnews.pro/news/ctok-reconstructed-claude-tokenizer.jsonld"}}