Accurate local Claude 4.7+ tokenizer

Sanderland's open-source ctok tool reconstructs Anthropic's Claude tokenizer, estimating 49,152 vocabulary entries in Claude 3 and 16,384 in Claude 4.7, with some reserved for image tokens. Claude 5's tokenizer has a vocabulary about 12 times smaller than OpenAI's o200k, resulting in 1.24x to 2.44x larger token counts on samples, and since Claude Opus 5 and GPT-5.6 Sol both charge $5 per million input tokens, the token ratio equals the cost ratio.

Reconstructing Claude's tokenizer ctok on GitHub ↗ https://github.com/sanderland/ctok Read the write-up ↗ /claude-tokenizer Claude publishes no tokenizer. count tokens hands back a number, one round trip at a time, and nothing about where it came from. ctok returns the same number offline, and the pieces behind it. Our best estimate of the vocabularies: 49,152 entries in v3, a third of that in v4.7 at 16,384, with some of both reserved for image tokens. Markers ride inside the piece they belong to: word edges, capitals, and raw bytes where the vocabulary has no piece for a character. The write-up /claude-tokenizer goes through them. The cost of a small vocabulary Claude 5's tokenizer comes at a cost, but less than its size suggests. Its vocabulary is about 12× smaller than o200k, while its token bill is 1.24× to 2.44× larger on these samples. Common pieces do most of the compression; a larger vocabulary spends the rest on rarer words and scripts. Claude Opus 5 and GPT-5.6 Sol both charge $5 per million input tokens, so the token ratio is the cost ratio.