Gigatoken: Fastest Tokenizer

Gigatoken, a new tokenizer implementation, claims to be the world's fastest, achieving speeds ~500-1000x faster than HuggingFace and ~100x faster than OpenAI's tiktoken on most machines, even though those baselines are multithreaded Rust implementations.

Introducing the world's fastest tokenizer implementation, Gigatoken Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines. These baselines are already multithreaded Rust implementations 🧵