Tokka-Bench

Researchers introduced Tokka-Bench, an open-source framework that evaluates tokenizers on five metrics — bytes per token, unique token coverage, subword fertility, word-split rate, and vocabulary comp…