WordPiece

Gigatoken, a Rust-based tokenizer by Marcel Rød, achieves 30x to 40x faster CPU tokenization than traditional implementations, with 37.6x speedup on a million-token request, by optimizing BPE encoding…

Researchers introduced QuechuaTok, a benchmark evaluating tokenization strategies for Southern Quechua, a low-resource agglutinative language. They found that BPE achieved the lowest fertility rate bu…