Tokka-Bench: Evaluating Tokenizers Across 100 Natural and 20 Programming Languages
Researchers introduced Tokka-Bench, an open-source framework that evaluates tokenizers on five metrics — bytes per token, unique token coverage, subword fertility, word-split rate, and vocabulary comp…