{"type": "article", "title": "Perplexity AI Open-Sources Unigram Tokenizer That Achieves 5x Lower p50 Latency Than Hugging Face tokenizers Crate", "publisher": "Web Pulse", "url": "https://wpnews.pro/news/perplexity-ai-open-sources-unigram-tokenizer-that-achieves-5x-lower-p50-latency", "original_source": "https://www.marktechpost.com/2026/05/28/perplexity-ai-open-sources-unigram-tokenizer-that-achieves-5x-lower-p50-latency-than-hugging-face-tokenizers-crate/", "published": "2026-05-28T09:08:20+00:00", "accessed": "2026-07-15", "id": "perplexity-ai-open-sources-unigram-tokenizer-that-achieves-5x-lower-p50-latency"}