04:00
2026-07-28
arxiv.org
natural-language-processing
BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis
Researchers have developed BHARATI, a set of SentencePiece BPE tokenizers for classical Indian languages, achieving a 90% reduction in sequence length compared to GPT-2 on a held-out test set of 490 I…