Tokenizer Expansion: Upgrading a Model's Tokenizer in Place
Liquid AI released LFM2.5-8B-A1B with a tokenizer expanded from 65K to 128K vocabulary, doubling the size to improve tokenization for Hindi, Vietnamese, and Thai without retraining from scratch. The n…