SentencePiece

A developer compares four subword tokenization algorithms—BPE, WordPiece, SentencePiece, and Unigram—used in major LLMs, explaining how tokenizer choice directly impacts inference cost, vocabulary cov…

Perplexity AI open-sourced a Unigram tokenizer reimplemented in Rust that achieves 5x lower p50 latency than the Hugging Face tokenizers crate at production input lengths. The new tokenizer, available…