Gigatoken

Gigatoken, a Rust-based tokenizer by Marcel Rød, achieves 30x to 40x faster CPU tokenization than traditional implementations, with 37.6x speedup on a million-token request, by optimizing BPE encoding…

Stanford PhD student Marcel Rød released Gigatoken, a Rust BPE tokenizer that encodes text at 24.53 GB/s on a 144-core AMD EPYC 9565 dual-socket setup, outperforming HuggingFace tokenizers by 989x and…

Gigatoken, a new tokenizer implementation, claims to be the world's fastest, achieving speeds ~500-1000x faster than HuggingFace and ~100x faster than OpenAI's tiktoken on most machines, even though t…