AdaptTok-Indic: Script-Aware Adaptive Tokenization for Indic Languages

A new arXiv paper (2610.33017v1) reports that AdaptTok-Indic, an Indic-specialized adaptive tokenizer, achieved mean token fertility of 1.432 for Bengali and 1.388 for Hindi across five recorded seeds, versus 2.156 and 2.085 for SentencePiece BPE. The tokenizer combines Unicode-normalized preprocessing, explicit Indic conjunct handling, script-aware routing and learned byte-level boundary prediction jointly trained with a small Transformer language model, and averaged 57.69 on downstream Indic evaluations against 50.82 for the SentencePiece BPE configuration. The authors describe the implementation as a small-scale prototype with documented simplifications and no statistical significance testing, released-MAGNET comparison or code-mixed evaluation.

arXiv:2610.33017v1 Announce Type: new Abstract: Multilingual tokenizers trained on English-dominated corpora often over-segment Indic languages, resulting in longer token sequences and increased tokenization disparity across scripts. We present AdaptTok-Indic, an Indic-specialized extension of adaptive tokenization that combines Unicode-normalized preprocessing with explicit Indic conjunct handling, script-aware routing, and learned byte-level boundary prediction jointly trained with a small Transformer language model. Evaluated on Bengali and Hindi, with additional evaluation on Gujarati, Tamil, and Telugu, AdaptTok-Indic achieves lower token fertility than SentencePiece BPE, an Indic-oriented tokenizer baseline, and an independently implemented MAGNET-style baseline. Across five recorded seeds, mean fertility is 1.432 for Bengali and 1.388 for Hindi, compared with 2.156 and 2.085 for SentencePiece BPE. Recorded downstream scores average 57.69 on evaluated Indic settings versus 50.82 for the SentencePiece BPE configuration. Independent verification runs using the released prototype confirm the same directional trends. The current implementation is a small-scale prototype with documented simplifications, and no statistical significance testing, released-MAGNET comparison, or code-mixed evaluation is performed. These results suggest that Indic-specialized adaptive segmentation can reduce tokenization disparity while maintaining competitive downstream performance.