01:19
2026-09-14
tokencontributions.substack.com
large-language-models
The Biology of Claude's Tokenizer
A developer has reverse-engineered Anthropic's Claude tokenizer, finding it is not a standard byte-level BPE but instead uses a minimum-piece tokenization scheme similar to MinGram or PathPiece. The r…