Gigatoken, a new tokenizer implementation, claims to be the world's fastest, achieving speeds ~500-1000x faster than HuggingFace and ~100x faster than OpenAI's tiktoken on most machines, even though those baselines are multithreaded Rust implementations.
Introducing the world's fastest tokenizer implementation, Gigatoken! Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines. These baselines are already multithreaded Rust implementations! 🧵
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.