09:08
2026-05-28
marktechpost.com
artificial-intelligence
Perplexity AI Open-Sources Unigram Tokenizer That Achieves 5x Lower p50 Latency Than Hugging Face tokenizers Crate
Perplexity AI open-sourced a Unigram tokenizer reimplemented in Rust that achieves 5x lower p50 latency than the Hugging Face tokenizers crate at production input lengths. The new tokenizer, available…