{"slug": "accurate-local-claude-4-7-tokenizer", "title": "Accurate local Claude 4.7+ tokenizer", "summary": "Sanderland's open-source ctok tool reconstructs Anthropic's Claude tokenizer, estimating 49,152 vocabulary entries in Claude 3 and 16,384 in Claude 4.7, with some reserved for image tokens. Claude 5's tokenizer has a vocabulary about 12 times smaller than OpenAI's o200k, resulting in 1.24x to 2.44x larger token counts on samples, and since Claude Opus 5 and GPT-5.6 Sol both charge $5 per million input tokens, the token ratio equals the cost ratio.", "body_md": "# Reconstructing Claude's tokenizer\n\n[ctok on GitHub ↗](https://github.com/sanderland/ctok)\n[Read the write-up ↗](/claude-tokenizer)\n\nClaude publishes no tokenizer. `count_tokens`\n\nhands back a\nnumber, one round trip at a time, and nothing about where it came\nfrom. ctok returns the same number offline, and the pieces behind it.\n\nOur best estimate of the vocabularies: 49,152 entries in v3, a third of that in v4.7 at 16,384, with some of both reserved for image tokens.\n\nMarkers ride inside the piece they belong to: word edges, capitals,\nand raw bytes where the vocabulary has no piece for a character.\n[The write-up](/claude-tokenizer) goes through them.\n\n## The cost of a small vocabulary\n\nClaude 5's tokenizer comes at a cost, but less than its size suggests. Its vocabulary is about 12× smaller than o200k, while its token bill is 1.24× to 2.44× larger on these samples. Common pieces do most of the compression; a larger vocabulary spends the rest on rarer words and scripts.\n\nClaude Opus 5 and GPT-5.6 Sol both charge $5 per million input tokens, so the token ratio is the cost ratio.", "url": "https://wpnews.pro/news/accurate-local-claude-4-7-tokenizer", "canonical_source": "https://www.tokenize.rs/claude", "published_at": "2026-08-14 02:35:14+00:00", "updated_at": "2026-08-14 03:10:55.968523+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "developer-tools"], "entities": ["Sanderland", "Anthropic", "Claude 3", "Claude 4.7", "Claude 5", "OpenAI", "o200k", "Claude Opus 5"], "alternates": {"html": "https://wpnews.pro/news/accurate-local-claude-4-7-tokenizer", "markdown": "https://wpnews.pro/news/accurate-local-claude-4-7-tokenizer.md", "text": "https://wpnews.pro/news/accurate-local-claude-4-7-tokenizer.txt", "jsonld": "https://wpnews.pro/news/accurate-local-claude-4-7-tokenizer.jsonld"}}