GigaToken: The Rust BPE Tokenizer That’s 989x Faster Than HuggingFace

On July 22, 2026, a Rust-based tokenizer called GigaToken, built by Marcel Rød, hit the top of Hacker News with claims of being 989x faster than HuggingFace tokenizers, processing text at 24.53 GB/s on a 144-core AMD EPYC and able to tokenize Common Crawl in under 7 hours. The speed gains come from replacing regex with SIMD instructions, caching pretoken-to-token mappings, and batching operations to minimize Python-Rust boundary crossings, though the optimizations are specific to BPE-based models and do not transfer to SentencePiece families like Llama or T5.

On July 22, 2026, a Rust-based tokenizer called GigaToken hit the top of Hacker News with a claim that stopped LLM engineers mid-scroll: 989x faster than HuggingFace tokenizers https://huggingface.co/docs/tokenizers/index . Marcel Rød’s one-person project processes text at 24.53 GB/s on a 144-core AMD EPYC — enough to tokenize the entirety of Common Crawl in under 7 hours. HuggingFace would need months. The 114-comment thread that followed wasn’t about whether the benchmarks were real. They are. The debate was about whether they matter. Why GigaToken Is That Fast: SIMD, Caching, Rust HuggingFace tokenizers run four steps on every input: normalize, pre-split, BPE merge, and post-process. Each step leans on regex engines. tiktoken is faster — roughly 2-3x — because it skips normalization and goes straight to a regex chunk before BPE. GigaToken replaces the regex entirely with SIMD instructions, processing multiple bytes in parallel at the CPU level. It also caches pretoken-to-token mappings. Token distributions in real text are highly long-tailed — a small set of pretokens covers most of what you’ll ever see — so caching turns repeated merge lookups from O k log k BPE operations into O 1 cache hits. Finally, GigaToken minimizes crossing the Python-Rust boundary by batching operations aggressively, eliminating the serialization overhead that other libraries pay thousands of times per document. The result: benchmarks on Apple M4 Max 16 cores show 8.79 GB/s for GPT-2, a 1,268x improvement over HuggingFace. The critical caveat is that these gains are specific to BPE-based models. SentencePiece families — Llama 1 and 2, T5, Gemma — use unigram tokenization rather than BPE, and GigaToken’s optimizations don’t transfer. Speedups on those families land in the low double digits rather than the hundreds. python import gigatoken as gt Drop-in replacement for HuggingFace tokenizers tokenizer = gt.Tokenizer "Qwen/Qwen3-8B" .as hf tokens = tokenizer.encode batch "Hello world", "Another text" When 989x Faster LLM Tokenization Actually Matters The HN skeptics had a point. For standard GPU inference, tokenization is under 0.1% of wall time. GPU VRAM transfers, prefill, attention computation, and decode dwarf it. Switching tokenizers in a typical inference service will not move your p99 latency in any measurable way. If that describes your workload, GigaToken is not for you right now. However, two categories of workloads change the calculation. First: pretraining and large-scale fine-tuning pipelines, where tokenization runs on CPU clusters before data ever reaches a GPU. At 24.53 GB/s, GigaToken turns batch preprocessing jobs from overnight processes into real-time operations. At this throughput on a 144-core EPYC, you can tokenize Common Crawl’s 130 trillion tokens in under 7 hours. That’s not a marginal improvement — that’s a different class of iteration speed. Second: large-context inference for agentic workflows. As prompts grow — tool call histories, retrieved code, conversation chains — tokenization costs climb. Rød published measurements showing up to 40% TTFT reduction for GPT-OSS-120B and 18% for DeepSeek V3 on large-context requests. For inference routing systems processing millions of requests a day, that compounds. The HN discussion thread https://news.ycombinator.com/item?id=49010167 captures this debate well — it’s worth reading if you’re evaluating GigaToken for production. Installation and Limitations Installation is a single command: pip install gigatoken The HuggingFace compatibility mode uses .as hf to return an object with an identical encode batch interface. Token output is byte-identical to HuggingFace Tokenizers in tested configurations — model weights and vocabulary are untouched. GigaToken 0.9.0 on GitHub https://github.com/marcelroed/gigatoken supports 23 tokenizer families including Llama 3 through 4, Qwen 2 through 3, DeepSeek V3 and V4, Phi-4, GLM 5, and GPT-2. The limitations are real. SentencePiece-based models see minimal gains. Windows support is incomplete WSL recommended . Version 0.9.0 is beta — not yet 1.0. File sinks aren’t implemented. And the benchmark methodology isn’t perfectly apples-to-apples: GigaToken encodes entire unsplit files; HuggingFace is benchmarked on 100MB samples. The 989x figure is a ceiling, not a floor. Install it, test it against your specific workload before committing to production use. The PyPI package page https://pypi.org/project/gigatoken/ has the full dependency and platform matrix. Key Takeaways - GigaToken is the fastest BPE tokenizer available, hitting 24.53 GB/s through SIMD pretokenization, pretoken caching, and minimal Python-Rust boundary crossings. - For pretraining data pipelines, GigaToken is a genuine unlock — turning multi-day tokenization jobs into hours. - For standard GPU inference, the speedup won’t move the needle. The math doesn’t favor switching unless you’re serving very large context windows at high volume. - Install with pip install gigatoken and migrate with the .as hf wrapper. SentencePiece-based models are excluded from significant gains. - Version 0.9.0 is beta. Test before committing, especially on non-x86 or Windows environments.