Tokencontributions (auto-discovered)

A developer has reverse-engineered Anthropic's Claude tokenizer, finding it is not a standard byte-level BPE but instead uses a minimum-piece tokenization scheme similar to MinGram or PathPiece. The r…

A developer's analysis of pre-tokenization regexes shows that GPT-2's word-splitting pattern omitted Unicode's Mark category, a bug inherited by GPT-4, Llama 3, Qwen 3, and GLM-4/5, forcing BPE to tok…