Claude's tokenizer remains a mystery, and with the recent 40%+ increase in token cost for basic English text, even more so. While everyone else ships a fairly standard byte-level BPE tokenizer, Anthropic seems to have placed their bets on something unique.
In this post I'll go through the most interesting properties I've found while reconstructing Claude's tokenizer.
Code is available here: https://github.com/sanderland/ctok
Claude’s tokenizer is not BPE #
A pairwise BPE construction requires every non-base token to be built from two smaller vocabulary pieces, so a one-token span must have at least one internal split into two things that are themselves tokens. That’s testable.
The Hangul syllable 최 (choi) is the first example I found. It is one token (which makes sense, a very common surname), but testing other Korean syllables which share a common UTF-8 prefix/suffix shows the 2-byte prefix and suffix are not tokens in any other character (i.e. all characters sharing a prefix or suffix cost 3, their length in bytes).
So what is it? The one rule that seems to consistently explain the token counts is a minimum-piece tokenization as in MinGram or PathPiece.
Vocabulary size #
My best estimate of vocabulary size is around 49-55k for Claude v3-v4.6 and around 16-20k after, based on the list of tokens found so far. (Update: it settled at 48k and 15k in the end, almost certainly 49,152 and 16,384 with some of those being reserved)
Boundary markers and spaces #
Boundary markers on words
One of the main novelties of the Claude tokenizer is wrapping word-like spans in boundary markers. I use ^ and $ as notation for these inferred begin- and end-of-word tokens. For example, “tokenizers” becomes [^token][izers$]
For example “semiconduct”, “usercontent” are two tokens and “romagnet” is three tokens in isolation, but the tokens here are [^semiconduct][$], [^][usercontent$] and [^][romagnet][$], that is, the phrases are tokens but just in the right context. And indeed, “semiconductromagnetusercontent” tokenizes to the expected three tokens.
Similarly, “telecommunications” is a single token, but “telecommunicationsy” is four ([^telecommun][ic][ation][sy$])
This also clarifies the conclusion of my previous post, where I suggested that most Claude tokens were whole words. What was really being measured was whether the fully marked form (^word$) was a one-token vocabulary entry.
Boundary markers on punctuation
Spans of punctuation characters also get markers, but only on the side where they border a space, e.g. the punctuation in a== b becomes ==$
Boundary markers eat spaces
At first sight, these markers may seem a little wasteful of vocabulary space, but this is compensated for by not needing different variants for word-with-a-space and word-without-a-space.
Specifically, I think that any space in $<space>^ is removed before encoding, and $^ becomes a space during decoding. This means that many whitespaces in code and normal text are “free”, and most of all, the model doesn’t split its training data between “space prefixed” and “non space” variants.
CapsCode #
Another relative novelty is how capitalization is handled. Much like space+word being a separate token feels redundant, so does having duplicate tokens for nasa, Nasa, and NASA.
For title-case and all-caps, Claude appears to use a shift marker or caps marker and then a lowercased form:
Token → ↑ token
NASA → ⇪ nasa
The marker itself can be part of a token, or be separate (just like ^ and $).
Notably, it only applies to a whole pretoken, not for every capital letter. Thus:
Token → marked ↑ token
NASA → marked ⇪ nasa
GaN → literal GaN
WiFi → literal WiFi
Thus, while MERCHANTABILITY is a single token [⇪^merchantability$], adding a lowercase letter “q” breaks both the caps-code and word marking, turning it into [^M][ER][C][H][A][N][T][A][B][I][L][I][T][Y][q$].
The newer v4.7+ tokenizer has simplified this a lot, and removes the caps-lock code.
Whitespace #
There are tokens for spans of spaces, newlines and tabs. Weirdly, 1,2,…,30,32,33,40,48,64,98,128 newlines are tokens, but 31 is not.
Message overhead #
There appears to be a fixed overhead of 7-8 tokens, such that “a” costs 8, but a single digit cost 9 tokens.
However, closer investigation suggests this is due to the surrounding chat formatting beginning with ^ and ending in \n\n (along with some other unknown content). In particular the ^ start can really lead to some puzzling results when trying to find out what is a token and what is not.
Odds and Ends #
- Han, Korean, emoji and many rare scripts do not have word markers, and are always tokenized character-by-character.
- For characters that don’t have a dedicated token, UTF-8 byte fallback is used, but only within a character, and strictly prefix-based (e.g. rather like suggested in SCRIPT-BPE ).
- Digits are handled with fairly normal groups-of-3.
- Normalization is NFC along with some character replacements (notably curly quotes in the earlier version).
Disclaimers and acknowledgements #
- All of the above is somewhat speculative, and none of it is from official sources.
- There appears to be some remaining unexplained behaviour in languages with many markers, e.g. Devanagari, where the error can be as large as 10% in counts.
- Most of the effort has been on the older version, v4.7+ has slightly larger errors.
This work was made possible in part due to:
- Earlier work by @sasuke___420 [1][2]
- The surprisingly large amount of token streams Anthropic published in the sources of Anthropic’s On the Biology of a Large Language Model
- Insights from Fable 5, particularly on the three CapsCode cases