Unigram

Unigram, a new Rust crate released on crates.io, encodes bytes as words that each cost exactly one LLM token, making an N-byte value cost exactly N tokens under Claude and other models. The bijective …

Perplexity AI open-sourced a Unigram tokenizer reimplemented in Rust that achieves 5x lower p50 latency than the Hugging Face tokenizers crate at production input lengths. The new tokenizer, available…