o200k

ByteLex reported that a tokenizer-free coordinate map built from 11 tokenizer vocabularies over 3-byte windows predicted its 237M-parameter byte-level language model's per-word errors with a Spearman …

Sanderland's open-source ctok tool reconstructs Anthropic's Claude tokenizer, estimating 49,152 vocabulary entries in Claude 3 and 16,384 in Claude 4.7, with some reserved for image tokens. Claude 5's…