The Biology of Claude's Tokenizer

A developer has reverse-engineered Anthropic's Claude tokenizer, finding it is not a standard byte-level BPE but instead uses a minimum-piece tokenization scheme similar to MinGram or PathPiece. The reconstruction reveals novel boundary markers that wrap word-like spans and a capitalization shift marker that lets the model encode title-case and all-caps words from lowercased forms, with the code released on GitHub.

Claude's tokenizer remains a mystery, and with the recent 40%+ increase in token cost for basic English text, even more so. While everyone else ships a fairly standard byte-level BPE tokenizer, Anthropic seems to have placed their bets on something unique. In this post I'll go through the most interesting properties I've found while reconstructing Claude's tokenizer. Code is available here: https://github.com/sanderland/ctok https://github.com/sanderland/ctok Claude’s tokenizer is not BPE A pairwise BPE construction requires every non-base token to be built from two smaller vocabulary pieces, so a one-token span must have at least one internal split into two things that are themselves tokens. That’s testable. The Hangul syllable 최 choi is the first example I found. It is one token which makes sense, a very common surname , but testing other Korean syllables which share a common UTF-8 prefix/suffix shows the 2-byte prefix and suffix are not tokens in any other character i.e. all characters sharing a prefix or suffix cost 3, their length in bytes . So what is it? The one rule that seems to consistently explain the token counts is a minimum-piece tokenization as in MinGram https://arxiv.org/abs/2606.27019 or PathPiece https://arxiv.org/abs/2402.18376 . Vocabulary size My best estimate of vocabulary size is around 49-55k for Claude v3-v4.6 and around 16-20k after, based on the list of tokens found so far. Update: it settled at 48k and 15k in the end, almost certainly 49,152 and 16,384 with some of those being reserved Boundary markers and spaces Boundary markers on words One of the main novelties of the Claude tokenizer is wrapping word-like spans in boundary markers. I use ^ and $ as notation for these inferred begin- and end-of-word tokens. For example, “tokenizers” becomes ^token izers$ For example “semiconduct”, “usercontent” are two tokens and “romagnet” is three tokens in isolation, but the tokens here are ^semiconduct $ , ^ usercontent$ and ^ romagnet $ , that is, the phrases are tokens but just in the right context. And indeed, “semiconductromagnetusercontent” tokenizes to the expected three tokens. Similarly, “telecommunications” is a single token, but “telecommunicationsy” is four ^telecommun ic ation sy$ This also clarifies the conclusion of my previous post https://tokencontributions.substack.com/p/whole-words-and-claude-tokenization , where I suggested that most Claude tokens were whole words. What was really being measured was whether the fully marked form ^word$ was a one-token vocabulary entry. Boundary markers on punctuation Spans of punctuation characters also get markers, but only on the side where they border a space, e.g. the punctuation in a== b becomes ==$ Boundary markers eat spaces At first sight, these markers may seem a little wasteful of vocabulary space, but this is compensated for by not needing different variants for word-with-a-space and word-without-a-space. Specifically, I think that any space in $<space ^ is removed before encoding, and $^ becomes a space during decoding. This means that many whitespaces in code and normal text are “free”, and most of all, the model doesn’t split its training data between “space prefixed” and “non space” variants. CapsCode Another relative novelty is how capitalization is handled. Much like space+word being a separate token feels redundant, so does having duplicate tokens for nasa, Nasa, and NASA. For title-case and all-caps, Claude appears to use a shift marker or caps marker and then a lowercased form: Token → ↑ token NASA → ⇪ nasa The marker itself can be part of a token, or be separate just like ^ and $ . Notably, it only applies to a whole pretoken, not for every capital letter. Thus: Token → marked ↑ token NASA → marked ⇪ nasa GaN → literal GaN WiFi → literal WiFi Thus, while MERCHANTABILITY is a single token ⇪^merchantability$ , adding a lowercase letter “q” breaks both the caps-code and word marking, turning it into ^M ER C H A N T A B I L I T Y q$ . The newer v4.7+ tokenizer has simplified this a lot, and removes the caps-lock code. Whitespace There are tokens for spans of spaces, newlines and tabs. Weirdly, 1,2,…,30,32,33,40,48,64,98,128 newlines are tokens, but 31 is not. Message overhead There appears to be a fixed overhead of 7-8 tokens, such that “a” costs 8, but a single digit cost 9 tokens. However, closer investigation suggests this is due to the surrounding chat formatting beginning with ^ and ending in \n\n along with some other unknown content . In particular the ^ start can really lead to some puzzling results when trying to find out what is a token and what is not. Odds and Ends - Han, Korean, emoji and many rare scripts do not have word markers, and are always tokenized character-by-character. - For characters that don’t have a dedicated token, UTF-8 byte fallback is used, but only within a character, and strictly prefix-based e.g. rather like suggested in SCRIPT-BPE https://arxiv.org/abs/2505.24689 . - Digits are handled with fairly normal groups-of-3. - Normalization is NFC along with some character replacements notably curly quotes in the earlier version . Disclaimers and acknowledgements - All of the above is somewhat speculative, and none of it is from official sources. - There appears to be some remaining unexplained behaviour in languages with many markers, e.g. Devanagari, where the error can be as large as 10% in counts. - Most of the effort has been on the older version, v4.7+ has slightly larger errors. This work was made possible in part due to: - Earlier work by @sasuke 420 \ 1\ https://x.com/sasuke 420/status/2034126531749142826 \ 2\ https://nitter.net/sasuke 420/status/1949932407219769799 m - The surprisingly large amount of token streams Anthropic published in the sources of Anthropic’s On the Biology of a Large Language Model https://transformer-circuits.pub/2025/attribution-graphs/biology.html - Insights from Fable 5, particularly on the three CapsCode cases