00:00
2026-07-30
blog.doubleword.ai
artificial-intelligence
The AI Industry Forgot How to Optimise a CPU
Gigatoken, a Rust-based tokenizer by Marcel Rød, achieves 30x to 40x faster CPU tokenization than traditional implementations, with 37.6x speedup on a million-token request, by optimizing BPE encoding…