Researchers introduced NE-BERT, a multilingual encoder model trained on 8.3 million sentences across 9 Northeast Indian languages plus Hindi and English, which outperforms IndicBERT-V2 and MuRIL on al…
Gigatoken, a Rust-based tokenizer by Marcel Rød, achieves 30x to 40x faster CPU tokenization than traditional implementations, with 37.6x speedup on a million-token request, by optimizing BPE encoding…
Researchers have developed BHARATI, a set of SentencePiece BPE tokenizers for classical Indian languages, achieving a 90% reduction in sequence length compared to GPT-2 on a held-out test set of 490 I…
A developer compares four subword tokenization algorithms—BPE, WordPiece, SentencePiece, and Unigram—used in major LLMs, explaining how tokenizer choice directly impacts inference cost, vocabulary cov…
Perplexity AI open-sourced a Unigram tokenizer reimplemented in Rust that achieves 5x lower p50 latency than the Hugging Face tokenizers crate at production input lengths. The new tokenizer, available…