Show HN: TriusLD – A Small Tokenizer for LLMs with Robust Multilingual Support

Trius released TriusLD, a tokenizer that natively covers 19 writing systems within a 65,535-token budget, shipping 64,972 tokens: 256 byte tokens, 14 special tokens and 64,702 merges. Trius reports TriusLD averages 0.52 tokens per character across 58 languages versus 0.84 for the vocabulary behind GPT-4, while English compression reaches 3.85 characters per token, roughly 18% behind a 200,000-token model. The design balances per-script quotas against a shared byte fallback so non-Latin scripts avoid costly UTF-8 byte-level segmentation.

TriusLD: A Small Tokenizer With Robust Multilingual Support Abstract. In this article, we introduce Trius's Little Dictionary TriusLD , a tokenizer that has native support for 19 writing systems within the budget of 65,535 token slots. By balancing the vocabulary between per-script quotas and a shared byte fallback, we managed to gain broad multilingual compression — 0.52 tokens per character averaged over 58 languages, against 0.84 for the vocabulary behind GPT-4<sup \ 2\ ref-2 </sup — without compromising the modelling of English too much: 3.85 characters per token, roughly 18% behind a 200,000-token model. 1 Introduction A tokenizer is the first thing a model reads and the last thing it writes, and it is the only part of the stack that decides what a token is worth. When the vocabulary is allocated by raw web frequency — which is what happens if you simply run BPE over a crawl — English and Chinese take almost everything, and every other writing system is left to fall back on raw bytes. That fallback is lossless but expensive. Under UTF-8 a Cyrillic, Greek, Hebrew or Arabic character is two bytes; a Han character or a Hangul syllable is three. With no merges to cover them, each character costs that many tokens. A Hindi sentence at three tokens per character is not merely inefficient, it is unusable: the context window shrinks by a factor of three and the model spends its attention budget reassembling letters. The obvious remedy is to spend vocabulary slots on the scripts. The trouble is that the vocabulary is a fixed budget, and every slot given to Unihan is a slot denied to English. Han alone could absorb twenty thousand characters; doing so would push English back toward byte-level fallback, which is exactly what we are trying to avoid. There is no allocation that is good for everyone, only a balance point. TriusLD is our answer to that balance problem, under a hard constraint of 65,535 slots — the largest vocabulary addressable by a 16-bit index. It ships 64,972: 256 byte tokens, 14 special tokens, and 64,702 merges. This article records how the slots were divided, why the obvious approach of training one BPE over everything does not work, and how the result compares against the tokenizers of several well-known models. 2 Methodology Token slots must be allocated to non-ASCII writing systems if byte-level fallback is to be avoided for them. Under UTF-8, Cyrillic, Greek, Hebrew and Arabic characters occupy two bytes each, while most Unihan characters occupy three; a script without coverage therefore costs two or three tokens per character. The allocation cannot be unbounded. CJK character sets alone may contain as many as 20,000 distinct characters, and every slot spent on them is a slot withheld from English, which regresses toward byte-level segmentation once its merges are starved. The design question is therefore where the balance is to be struck. 2.1 The arithmetic of the budget Three things are not negotiable before any script gets a slot. The first is the 256-token byte alphabet: it is what makes the tokenizer total, in the sense that any byte string whatsoever — a script we never planned for, a corrupted file, an emoji sequence — can be encoded and decoded without loss. The second is the special-token block: beginning and end of sequence, padding, chat roles, fill-in-the-middle markers, and a small reservation for tools. The third is the character inventory of each script, which is discussed below and which turned out to be the largest single line item after Han. What remains is spent on merges, and merges are where the trade is made. A merge that turns two Cyrillic characters into one token saves a token on every occurrence of that pair; a merge that turns th into a token saves a token on every th in English. These are directly comparable, and in a frequency-driven trainer English wins almost every time. 2.2 Latin, and the scripts with many national variants Many languages use the Latin script with accent marks — especially Eastern European languages and Vietnamese, which has a great many of them. Since the Unicode standard includes both “combined” characters and “combining” characters e.g. á versus a + ´ , we decided to enforce NFC. NFC matters more than it looks. Without it, the same Vietnamese word has two encodings, the merges learned for one do not apply to the other, and the character inventory doubles. Normalising to NFC at both training and inference time means the vocabulary only has to hold the precomposed forms — about four hundred characters for all of Western and Eastern Europe plus Vietnamese. The few sequences that NFC cannot compose a base letter carrying both a tone and a diacritic are covered by the combining marks that survive normalisation. We deliberately do not apply NFKC. It would fold fullwidth punctuation to ASCII, turn fi into fi , and quietly destroy information that the model may later need to reproduce. A tokenizer that cannot round-trip its input is a lossy compressor that has been placed in front of the loss function. 2.3 Unihan There are technically about 60,000 Han glyphs, but a majority of them occur only rarely e.g. in old Chinese literature . Most running text can be covered with a subset of about 10,000 to 13,000 characters: the first 3,000 cover roughly 99% of modern Chinese, and the tail past 13,000 is under a hundredth of a percent. We decided to include only the more commonly seen characters, together with a few thousand Han character bigrams. Two complications. First, Simplified and Traditional Chinese are different code points, not normalisation variants; they cannot be folded together, and both inventories must be carried. Second, Japanese kanji overlap heavily with Traditional Chinese but not entirely, so the union of the three sets is larger than any one of them. We ended up with roughly 13,000 single characters, which is a meaningful fraction of the whole vocabulary and the single largest item in the budget. 2.4 Hangul Depending on what one means by “Hangul support”, this could be anything between about 2,350 slots and 11,172. The Unicode syllable block contains every legal modern combination; the 2,350 syllables of KS X 1001 are what actually occurs in text, and cover better than 99.9% of it. We took the smaller set, plus the jamo, so that a rare syllable degrades to two or three tokens by composition rather than six or nine by bytes. The eight thousand slots this frees are worth more elsewhere: because those syllables account for 99.9% of running text, the remainder is drawn from such a thin tail that composing it from jamo costs almost nothing in expectation, whereas the same slots buy a measurable gain in a Latin inventory that has to serve fourteen languages. 2.5 The South Asian scripts These are the awkward ones, for a reason that has nothing to do with size. There are ten of them in our set — Devanagari, Bengali, Tamil, Telugu, Malayalam, Gujarati, Kannada, Odia, Gurmukhi, Sinhala — and they share structure but not code points. Unlike French, Spanish and German, which share both alphabet and a large number of merges -tion , -mente , -ung , a Bengali merge is worth nothing to Tamil. Each script is a separate budget line with no cross-subsidy. The one piece of good news is that the base characters are cheap. Covering all ten scripts takes about 800 slots; that alone is the difference between three tokens per character and one, which is the highest return available anywhere in the vocabulary. The merges above that are where it gets expensive, and where we chose to be selective. 2.6 The corpus Training text came from FineWeb for English and FineWeb-2 for everything else, fetched through a public mirror<sup \ 6\ ref-6 </sup . We keep one UTF-8 text file per language, one document per line, 58 files, 6.16 GB in total. Volumes are deliberately unequal: 600 MB of English, 250 MB each for the other large languages, and as little as 15 MB for the smallest Cyrillic ones. A BPE merge distribution stabilises long before hundreds of megabytes, so the minor languages do not need an equal share. 3 Decisions 3.1 The allocation The final division of the merge budget. The characters column is the number of most-frequent characters that are forced to be single tokens; the quota column is what the group's own BPE training was allowed to spend. Special tokens, the byte alphabet, and the coverage top-up sit outside this table. | group | scripts | quota | characters | |---|---|---|---| | latin | Latin | 22,500 | 4,000 | | han | Han S + T + J | 13,000 | 13,000 | | cyrillic | Cyrillic | 3,300 | 1,000 | | arabic | Arabic | 2,900 | 1,000 | | hangul | Hangul | 2,800 | 2,350 | | kana | Hiragana, Katakana | 1,400 | 400 | | devanagari | Devanagari | 1,200 | 600 | | greek | Greek | 900 | 600 | | thai | Thai | 800 | 400 | | hebrew | Hebrew | 700 | 400 | | bengali | Bengali | 600 | 500 | | tamil | Tamil | 500 | 400 | | telugu | Telugu | 500 | 400 | | odia | Oriya | 400 | 300 | | gurmukhi | Gurmukhi | 400 | 300 | | sinhala | Sinhala | 300 | 300 | | malayalam | Malayalam | 220 | 400 | | gujarati | Gujarati | 180 | 300 | | kannada | Kannada | 180 | 300 | The share of Latin is 35% — far below the 80% that an English-centric tokenizer would spend, which is precisely where the English cost comes from. Han takes another 20%. Everything else, nineteen scripts and fifty-odd languages, lives on 45% of the merges. 3.2 Route by script, not by language The first attempt trained one BPE over the whole corpus. That does not work, and not for a subtle reason: merges are selected by frequency, so the allocation follows the corpus rather than the plan. If the crawl is 40% English, English gets 40% of the merges regardless of what the table above says. We therefore split the corpus by script and train one BPE per group with an explicit quota<sup \ 7\ ref-7 </sup . Because every group shares the same 256-token byte alphabet, the resulting merge lists are composable: a merge is just a pair of byte-level tokens, and the pairs from independent trainings do not conflict. Routing is done per line , by the set of letter scripts the line contains. A monolingual line goes to its script group whole; a genuinely mixed line is split run by run. The detail that matters is where spaces and punctuation go: they follow the line's dominant script rather than being dumped into a shared bucket. That is what allows the Latin BPE to learn ␣the as one token. An earlier version routed every space to a shared group, and the Latin model could never see a leading space at all. 3.3 Force the characters Training alone does not guarantee that a character is a single token. A rare Han character appearing four hundred times in sixty million will not earn a merge, and will still cost three tokens — which quietly defeats the point of including it. After training, for each group we take the top N characters by frequency and synthesise the minimal merge chain that makes each of them a single token, sharing prefixes so that the cost is about one slot per character rather than two. Those merges are marked protected: the budget trimmer is not allowed to remove them. In the final build this step contributed 12,256 merges, almost all of them Han — which is the honest measure of how much the trainer missed on its own. 3.4 Compose the merge lists carefully Composition sounds mechanical and is not. Four things went wrong, and three of them were only visible as a quality regression after the whole vocabulary was assembled. Ordering. BPE applies merges by rank, so the relative order of two merges is part of the model. Our first union sorted merges by a global frequency estimate, which interleaved the scripts. That is harmless until a script's merge depends on a token produced by another script's merge — and it does, because every group learns its own merges for spaces, digits and stray Latin. The consequence was that a Cyrillic merge could be ranked after the token it needs, and the whole chain was applied out of order. Russian measured 0.66 tokens per character. A standalone Cyrillic model on the identical data measured 0.47. Ownership. The cause turned out to be deduplication. A merge pair learned by several groups was attributed to the first group that produced it. The Latin corpus contains a little stray Cyrillic — names, quoted text — so Latin learned ten foundational Cyrillic byte merges and owned them at a late rank. Cyrillic's identical pairs were then dropped as duplicates, and Cyrillic could never build its own characters early. Each pair now keeps the earliest rank any group assigns it. Russian went from 0.66 to 0.46 in one change, and Arabic, Persian and Urdu recovered with it. Long pre-tokens. BPE training is superlinear in the length of a single pre-token. A scraped page whose newlines had been stripped arrived as one 2.8-million-character “word”; Cyrillic training ran for twenty-five minutes without finishing. Lines are now broken every 4,000 characters during the split, and the no-space scripts every 500. Han training went from twenty-five minutes to nine. Byte caps are characters. A cap expressed in bytes admits twice as much Cyrillic as it does English, because a Cyrillic character is two bytes. This is obvious in hindsight and cost us an afternoon of swapping. 3.5 What we did not do We did not include Bengali, Tamil or the other South Asian scripts at full strength. We did not include the remaining Indic scripts at all — Telugu, Kannada and the rest stop at base characters plus a few hundred merges, and there is no coverage for Sinhala beyond its inventory. We did not add code-specific merges; code is served by the Latin group, which is adequate rather than good. We did not use a unigram model, byte-pair dropout, or any of the more elaborate segmentation schemes, on the grounds that a tokenizer nobody can debug is worse than one that loses a few percent. 4 Results 4.1 Tokens per character Measured on held-out text: the FineWeb-2 test split<sup \ 6\ ref-6 </sup , 100,000 characters per language, which the tokenizer never saw. Lower is better; tpc is tokens per character. | language | tpc | char/tok | language | tpc | char/tok | |---|---|---|---|---|---| | English | 0.260 | 3.85 | Hindi | 0.657 | 1.52 | | Indonesian | 0.275 | 3.64 | Japanese | 0.678 | 1.48 | | Vietnamese | 0.276 | 3.62 | Nepali | 0.707 | 1.41 | | Spanish | 0.277 | 3.61 | Chinese S | 0.708 | 1.41 | | Italian | 0.290 | 3.45 | Bengali | 0.722 | 1.39 | | German | 0.293 | 3.41 | Uyghur | 0.725 | 1.38 | | French | 0.293 | 3.41 | Assamese | 0.729 | 1.37 | | Dutch | 0.294 | 3.40 | Korean | 0.733 | 1.36 | | Portuguese | 0.295 | 3.39 | Sinhala | 0.736 | 1.36 | | Turkish | 0.310 | 3.23 | Odia | 0.739 | 1.35 | | Polish | 0.327 | 3.06 | Gujarati | 0.742 | 1.35 | | Russian | 0.461 | 2.17 | Tamil | 0.754 | 1.33 | | Greek | 0.479 | 2.09 | Telugu | 0.757 | 1.32 | | Ukrainian | 0.493 | 2.03 | Kannada | 0.779 | 1.28 | | Arabic | 0.522 | 1.92 | Malayalam | 0.801 | 1.25 | | Urdu | 0.544 | 1.84 | Chinese T | 0.802 | 1.25 | | Hebrew | 0.556 | 1.80 | Serbian | 0.476 | 2.10 | | Thai | 0.595 | 1.68 | Bulgarian | 0.447 | 2.24 | Family averages over all 58 languages: Latin 0.299, Cyrillic 0.494, Arabic script 0.519, Greek and Hebrew 0.540, Thai 0.595, CJK 0.730, Indic 0.732. 4.2 Against other tokenizers The same held-out text, encoded by the tokenizers of several well-known models. “Average” is the unweighted mean over 58 languages; “web” weights them by a rough share of web text English 40%, CJK 18%, Indic 13%, and so on , which is closer to how a general model is actually used. Both are tokens per character. | tokenizer | vocab | average | web | EN | |---|---|---|---|---| | TriusLD | 64,972 | 0.524 | 0.422 | 0.260 | | GPT-2 <sup \ 1\ ref-1 </sup | 50,257 | 1.264 | 0.859 | 0.220 | | cl100k GPT-4 <sup \ 2\ ref-2 </sup | 100,277 | 0.842 | 0.564 | 0.215 | | Llama 3 <sup \ 3\ ref-3 </sup | 128,256 | 0.699 | 0.446 | 0.215 | | Qwen3 <sup \ 4\ ref-4 </sup | 151,643 | 0.675 | 0.443 | 0.220 | | o200k GPT-4o <sup \ 2\ ref-2 </sup | 200,019 | 0.394 | 0.338 | 0.213 | | Gemma 3 <sup \ 5\ ref-5 </sup | 262,144 | 0.392 | 0.319 | 0.219 | Read plainly: at 65,000 slots TriusLD is closer to the 200,000-token models than to the 100,000-token ones. It beats cl100k, Llama 3 and Qwen3 on multilingual breadth despite having between a half and two-fifths of their vocabulary, and loses to o200k and Gemma 3, which have three and four times as many slots. On English it is about 20% behind all of them — the price of spending 35% of the vocabulary on Latin rather than 80%. The family breakdown is more informative than the average. | family | TriusLD | cl100k | Llama 3 | Qwen3 | o200k | Gemma 3 | |---|---|---|---|---|---|---| | Latin | 0.299 | 0.327 | 0.295 | 0.309 | 0.265 | 0.264 | | Cyrillic | 0.494 | 0.678 | 0.544 | 0.553 | 0.381 | 0.428 | | Arabic | 0.519 | 0.856 | 0.596 | 0.611 | 0.404 | 0.415 | | Indic | 0.732 | 1.469 | 1.358 | 1.261 | 0.434 | 0.383 | | CJK | 0.730 | 1.133 | 0.755 | 0.704 | 0.751 | 0.649 | | Greek, Hebrew | 0.540 | 0.967 | 0.796 | 0.632 | 0.380 | 0.454 | Chinese is the result we are least embarrassed by. At 0.708 for Simplified and 0.802 for Traditional we are within four percent of Qwen3 — a tokenizer with 2.3 times our vocabulary and an obvious Chinese bias — and ahead of Llama 3 on both, which has twice the slots. Indic is the result we are most embarrassed by. We put 2,660 merges into ten scripts and reached 0.732; Gemma 3 reaches 0.383 and GPT-4o 0.434. That is not a rounding difference, it is a different class of tokenizer. Ten unrelated scripts, none of which share a code point, is simply an expensive thing to do properly, and we did it at about a third of the necessary budget. 4.3 What the numbers do not say Our evaluation text is the held-out split of the same corpus the tokenizer was trained on, so the figures are mildly favourable to TriusLD: it has an in-domain advantage that the other models do not. We would expect a few percent, not a category change, but it should be kept in mind when reading the averages. Token counts are also not the same as model quality. What we can say is narrower: for the same text, TriusLD produces fewer tokens across a wider set of languages than a comparably-sized vocabulary, and slightly more in English. Whether that translates into a better model is a separate question. 4.4 What the vocabulary costs the model A tokenizer is usually argued about as a question of data efficiency — how many tokens a document costs — but the vocabulary is also a structural parameter of the model, and it appears in three places in the training budget. The input embedding and the output projection each cost V × d parameters. The projection itself costs about 6 · d · V FLOPs per token once the backward pass is counted. And the logits it produces are an activation of shape batch × sequence × V : at a 200,000-token vocabulary and a 4,096-token context that is 3.3 GB in fp32 for a single sequence, against 1.1 GB for 65,000. The relative weight of all three falls as the model grows, and the ratio is simple enough to carry around: the vocabulary's share of training compute is roughly V / 12 · L · d for a model of L layers and width d . A 200,000-token vocabulary is about 8% of training FLOPs at 32 layers and width 4,096, and about 65% at 20 layers and width 1,280. Vocabulary size is therefore close to free for large models and close to prohibitive for small ones, which is why frontier tokenizers push to 200,000 and 262,144 while a half-billion-parameter model cannot afford them. The other side of the ledger is sequence length, and it is where a compact vocabulary pays. Needing more tokens for the same text means a longer sequence, which costs attention quadratically: 20% more tokens per character is roughly 44% more attention per character, though attention is only part of the per-token cost, so the realised penalty is smaller than that. It also means more optimiser steps to consume the same corpus, so at a fixed compute budget a heavier tokenizer simply sees less text, and a fixed context window holds fewer characters. Two smaller effects point the same way: in a large vocabulary the tail tokens are updated so rarely that their embeddings stay near initialisation, and a finer segmentation asks the model to spend capacity on composition that a coarser one would have folded into the token. None of this changes the information content of the data — bits per byte is a property of the text, not of the tokenizer — only the compute and the parameter count required to extract it. 5 Conclusion TriusLD is not a good tokenizer in the sense of being better than a 200,000-token one. It is a good tokenizer in the sense of being nearly as good, at a third of the size, across a much wider set of languages than its peers bother with. The mechanism is not clever: divide the budget explicitly, route the corpus by script, force the characters, and do not let one group's frequency decide another group's merges. Two things we would do differently with a larger budget. The first is Indic: doubling those quotas and paying for it out of Han bigrams is the single highest-return change available, because our Chinese has slack that our Tamil does not. The second is English: at 0.260 we are leaving about 18% on the table, and a 100,000-slot version could recover most of it without giving up any breadth. The tokenizer is published at TriusAI/TriusLD https://huggingface.co/TriusAI/TriusLD and loads with AutoTokenizer.from pretrained "TriusAI/TriusLD" . The build script and the evaluation harness are reproducible from the same 6 GB of text in under two hours on a single machine. — Z Lin References 1. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI, 2019. GPT-2 tokenizer. 2. OpenAI. tiktoken : cl100k base and o200k base byte-pair encodings. 2022–2024. 3. AI@Meta. The Llama 3 Herd of Models. 2024. 4. Qwen Team. Qwen3 Technical Report. 2025. 5. Gemma Team, Google DeepMind. Gemma 3 Technical Report. 2025. 6. HuggingFace. FineWeb and FineWeb-2 datasets. 2024–2025. 7. HuggingFace. tokenizers : fast tokenizers for research and production. 2019–2025. ❧