Engineering Today, we're sharing the recipe behind the new tokenizer in LFM2.5-8B-A1B. It upgrades a pre-trained model's tokenizer in place, without retraining from scratch. We doubled the vocabulary from 65K to 128K to fix the languages our original tokenizer split too finely. Hindi and Vietnamese now take roughly 2.4× and 2.6× fewer tokens, and Thai up to 4.0× fewer, which we estimate makes per-character decoding 2.2 to 3.7× faster for these languages on-device. The quality of the languages the model already handled well holds steady.
LFM2.5-8B-A1B is available on Hugging Face, and the expanded 128K tokenizer is released alongside it. The full method, ablations, and per-device numbers are in the technical report.
Why on-device tokenizers stay small #
A tokenizer is fixed at the start of pre-training, and it splits the vocabulary in proportion to whatever the training corpus looked like then. Languages that were underrepresented at that point get broken into far more tokens per word than the ones the tokenizer was tuned for [1, 2]. Because a model runs its decoder once per output token, that extra fragmentation costs real latency, compute, and energy for users of those languages.
Cloud models can hide this cost behind a large vocabulary, since the embedding and output (LM-head) matrices are a small fraction of their parameters. On a small on-device model, they are not. At batch size 1, which is what we often care about on-device, decoding is limited by memory bandwidth, and the LM-head reads the whole vocabulary on every step. A larger vocabulary means a larger matrix to stream on each token and to keep in RAM, so edge models ship compact vocabularies and live with fragmentation outside their priority languages.
LFM2's original 65K byte-level Byte Pair Encoding (BPE) [3, 4] tokenizer was built for English, code, and a fixed set of languages, leaving little budget for Hindi, Vietnamese, or Thai. We wanted to fix that on a checkpoint we had already trained. Retraining the tokenizer and pre-training again would throw away that compute, and dropping in an unrelated third-party tokenizer would force us to solve a harder transfer problem than we needed to [5].
The recipe: extend, don't replace #
Our tokenizer expansion recipe has two parts: building the new tokenizer, then adapting the model to it. Because we design the tokenizer ourselves, we can build the new one as a direct extension of the old one, which keeps everything downstream simple.
Figure 1. The tokenizer expansion pipeline. We extend the source tokenizer's BPE merges to build the 128K vocabulary (top), reinitialize the model's embeddings from it, and adapt the model in two stages before mid-training and post-training (bottom).
1. Continue the BPE merges
A BPE tokenizer is defined by an ordered list of merge rules that combine smaller tokens into larger ones. For our recipe, we seed the new tokenizer's merge table with the original merges, freeze them, and continue BPE training on a multilingual corpus. This gives us two useful properties. First, most of the original 65K tokens carry over unchanged, so their learned representations transfer directly. Second, every new token has an exact decomposition into a sequence of original tokens.
2. Initialize the new embeddings from what the model already knows
This exact decomposition makes it easy to initialize the new token’s embeddings. Tokens that carry over keep their original embedding row. Each new token gets the mean of its sub-tokens' rows. Nothing is initialized randomly, and there is no cross-tokenizer alignment to work out. Since LFM2.5-8B-A1B ties its input and output embeddings, one matrix covers both.
3. Adapt the model in two stages
Training every parameter at once degraded the parts of the model that already worked, so we split adaptation into two stages, both run before the usual mid-training and post-training:
Stage 1, embedding only. Train only the new rows on 600B tokens with the rest of the model frozen. The new tokens settle without disturbing the body, and this alone recovers most of the quality lost at the swap.Stage 2, full continued pre-training. Unfreeze everything and continue pre-training on 400B tokens of a balanced multilingual mixture. This folds the new vocabulary into the body of the model and closes the remaining gap.
After stage 2, the checkpoint drops straight into the standard LFM2.5 pipeline with no other changes.
What the new tokenizer buys #
Figure 2. Encoding-length reduction of the expanded tokenizer over the source, per language (source token count ÷ expanded token count on the same text). Higher is better. 1.0× is parity.
The largest reductions are on Thai (4.0×), Bengali (3.4×), Vietnamese (2.6×), and Hindi (2.4×). Figure 3. The sentence "The train is running ten minutes late." encoded by the source 65K tokenizer and the expanded 128K tokenizer. Each box is one token, and every token is one decoder step. A character that spans several adjoining boxes is split by the source tokenizer into that many byte-level tokens. The reduction shown next to each language is for this sentence. Corpus-level averages are 2.4× for Hindi, 3.4× for Bengali, 2.6× for Vietnamese, and 4.0× for Thai.
Fewer tokens, faster decoding #
character, not per token. Putting the compression together with that per-token cost gives the net result below.
Figure 4. Synthesized decode speedup per character of generated output (M4 Max GPU shown, the other devices are within a few percent). Values above 1.0× mean the same text is produced faster with the expanded tokenizer.
This is the trade we designed for. Under-served scripts get large speedups, and languages the old tokenizer already handled well pay a small, uniform per-token cost that shows up as a decode regression of up to about 9%. For the languages and devices we target, that trade is well worth it. We stopped at 128K because pushing the vocabulary higher costs far more throughput on phones (up to 37% at 256K on the Snapdragon) than the extra compression would yield.
Quality holds up #
Figure 5. Aggregate quality (unweighted mean of eight benchmarks) across the adaptation stages. The zero-shot swap loses about 6 points, stage 1 recovers most of it, and stage 2 closes the gap and passes the source.
The recovery is steady. The swap costs 5.8 aggregate points, stage 1 wins back 4.8 of them, and stage 2 recovers the rest and passes the source. The surplus over the source most likely reflects the extra 400B tokens of pre-training in stage 2 rather than the vocabulary change itself. What matters is that the expanded model holds or improves on knowledge, math, code, and multilingual benchmarks while using the 128K tokenizer.
Per-language Global-MMLU [6] shows both halves of the goal at once. The languages we already supported hold steady, and the ones we under-tokenized gain:
Figure 6. Global-MMLU accuracy (5-shot, chance = 25%). Under-tokenized languages see large gains. Previously supported languages hold steady.
Where this applies #
This recipe fits a specific case: you own the tokenizer and can continue its original BPE merges, which needs the original merge rules and special-token configuration. It does not help when you are moving to an off-the-shelf third-party tokenizer, where zero-shot transfer methods are the right tool [5]. Stage 2 also costs some continued pre-training compute, so the approach pays off most when the original training run was expensive enough that reusing it beats starting over.
Get started #
LFM2.5-8B-A1B is released with open weights on Hugging Face under the LFM Open License v1.0, with deployment guides for llama.cpp, MLX, vLLM, and SGLang. The expanded 128K tokenizer is released alongside the weights. You can find the full method, ablations, and per-device numbers in the technical report.
Citation #
If you use this work, please cite the technical report:
References #
[1] Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. (2023). Language model tokenizers introduce unfairness between languages. NeurIPS. https://arxiv.org/abs/2305.15425
[2] Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. (2021). How good is your tokenizer? On the monolingual performance of multilingual language models. ACL-IJCNLP. https://arxiv.org/abs/2012.15613
[3] Rico Sennrich, Barry Haddow, and Alexandra Birch. (2016). Neural machine translation of rare words with subword units. ACL. https://arxiv.org/abs/1508.07909 [4] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. (2019). Language models are unsupervised multitask learners. OpenAI Technical Report. https://shorturl.at/hHoQR
[5] Benjamin Minixhofer, Edoardo M. Ponti, and Ivan Vulić. (2024). Zero-shot tokenizer transfer. NeurIPS. https://arxiv.org/abs/2405.07883 [6] Shivalika Singh et al. (2024). Global MMLU: Understanding and addressing cultural and linguistic biases in multilingual evaluation. arXiv. https://arxiv.org/abs/2412.03304