Quicktok: A faster tokenizer
Developer dmatth1 released quicktok, a fast exact BPE tokenizer written in C++ that runs 2–3.5× faster than bpe-openai and 4–11× faster than tiktoken on CPU. The tokenizer is byte-identical to tiktoken and available as a…