A developer shared a Python script that extracts a SentencePiece tokenizer model from GGUF files, enabling the use of the tokenizer with the sentencepiece library. The approach reads token, score, and…
A developer's guide explains that local token counting for Gemini 3.8 Flash using tiktoken or Hugging Face tokenizers is inaccurate, leading to budget overruns and API errors. The recommended solution…
Researchers introduced NE-BERT, a multilingual encoder model trained on 8.3 million sentences across 9 Northeast Indian languages plus Hindi and English, which outperforms IndicBERT-V2 and MuRIL on al…
Gigatoken, a Rust-based tokenizer by Marcel Rød, achieves 30x to 40x faster CPU tokenization than traditional implementations, with 37.6x speedup on a million-token request, by optimizing BPE encoding…
Researchers have developed BHARATI, a set of SentencePiece BPE tokenizers for classical Indian languages, achieving a 90% reduction in sequence length compared to GPT-2 on a held-out test set of 490 I…
A developer compares four subword tokenization algorithms—BPE, WordPiece, SentencePiece, and Unigram—used in major LLMs, explaining how tokenizer choice directly impacts inference cost, vocabulary cov…
Perplexity AI open-sourced a Unigram tokenizer reimplemented in Rust that achieves 5x lower p50 latency than the Hugging Face tokenizers crate at production input lengths. The new tokenizer, available…