[ctok on GitHub ↗](https://github.com/sanderland/ctok)
[Read the write-up ↗](/claude-tokenizer)
Claude publishes no tokenizer. count_tokens
hands back a number, one round trip at a time, and nothing about where it came from. ctok returns the same number offline, and the pieces behind it.
Our best estimate of the vocabularies: 49,152 entries in v3, a third of that in v4.7 at 16,384, with some of both reserved for image tokens.
Markers ride inside the piece they belong to: word edges, capitals, and raw bytes where the vocabulary has no piece for a character.
The write-up goes through them.
The cost of a small vocabulary #
Claude 5's tokenizer comes at a cost, but less than its size suggests. Its vocabulary is about 12× smaller than o200k, while its token bill is 1.24× to 2.44× larger on these samples. Common pieces do most of the compression; a larger vocabulary spends the rest on rarer words and scripts.
Claude Opus 5 and GPT-5.6 Sol both charge $5 per million input tokens, so the token ratio is the cost ratio.