GigaToken

On July 22, 2026, a Rust-based tokenizer called GigaToken, built by Marcel Rød, hit the top of Hacker News with claims of being 989x faster than HuggingFace tokenizers, processing text at 24.53 GB/s o…

GigaToken, a new open-source library, claims to be up to 1000x faster than HuggingFace's tokenizers, achieving 24.53 GB/s on GPT-2 tokenization on a server CPU. The library is a drop-in replacement fo…

GigaToken, a new open-source tokenizer, claims to be up to ~1000x faster than HuggingFace's tokenizers and tiktoken for language model tokenization, achieving speeds of over 24 GB/s on an AMD EPYC 956…