{"slug": "tokenisation-who-decides-what-a-token-is-anyway", "title": "Tokenisation: Who decides what a token is anyway?", "summary": "Tokenisation, the process of converting text into discrete tokens for neural networks, is a fixed preprocessing step that shapes what language models can learn, yet it is often overlooked because it does not appear in the training loss. The choice between character-level and word-level tokenisation involves trade-offs in vocabulary size and sequence length, leading to the common use of sub-word tokenisation. The article explains that the tokeniser's fixed nature means models cannot update token boundaries during training, which can affect tasks like counting letters in words, as seen with the 'strawberry' example.", "body_md": "Who decides what a token is anyway?\n\nHow does a neural network understand text? It's a question so basic, you can be forgiven for forgetting about it. Words and letters are human-readable, but are discrete (even if infinite in combination) and not compatible with the continuous representations that neural networks use. To bridge the gap, we need to take a sequence of human readable information and quantise it into a fixed set of pre-defined objects. This process is called tokenisation.\n\nA tokeniser works on integer sequences. e.g. `\"the cat sat\"`\n\n→ `[1169, 3797, 3332]`\n\nso the text can be mapped into a fixed vocabulary, the actual embeddings come after.\nThe tokeniser itself is a separate model, usually trained separately from the core language model, and is fixed before pretraining. It is then inherited by the model as it learns, which makes the choice of how we tokenise the text one of a very few decisions you can't update run to run.\n\nNo matter how long the training takes, the model can’t update the initial tokenised representation just from the training loss, just the embedding of each individual token.\n\nThe slightly undervalued or forgotten place of the tokeniser is exactly because it doesn't feature directly in the loss curve, it’s not a parameter you typically think about when it comes to tuning your model. But it defines what's easy or hard to learn and that alone is reason to understand where the artificial boundaries come from.\n\nWe've all seen the classic example: \"how many times does the letter `r`\n\nappear in strawberry?\" For a long time models were replying “two”. The usual explanation is that the model sees tokens `str/aw/berry`\n\n(as in cl100k), and not the individual letters, so it can't count. This is true but a bit shallow (just see the current models).\n\nThis was ultimately fixed by using a reasoning model's chain of thought, which spells out the word and has an easier time counting the occurrences. But the fact the model couldn't do this without a reasoning harness is a direct consequence of the tokeniser.\n\nWhen we think about how to split up the text into chunks there are two natural ways to divide the text; splitting each word, or splitting each character.\n\nIf we start with the character level split, it feels obvious because we know there are only 26 letters in the English language, we can double that if we want to include capital letters, 10 numbers, a bunch of punctuation, and then even if we want to expand to Romance languages the full set of characters needed including all accented letters is measured in the low hundreds. (Deliberately avoiding non-Romance languages for now.)\n\nThe beauty of this simplicity is that it gives us a tiny vocab, and we can be confident that we never encounter anything unknown. The cost we pay for this simplicity is in the length of the sequence, a passage containing a few hundred characters requires the sequence to be a few-hundred tokens as well. If we are modelling this with attention, we need to remember it scales with sequence length squared. This granularity is expensive.\n\nThe other end of this spectrum is to divide the sequence into individual words. As people we have intuition about the definition of words, so this feels like another natural option. This gives us much shorter sequences, where the meaning associated with each token must be much denser. This seems like it could be a good property, but it requires a vocabulary that grows essentially unbounded. And more than that follows a characteristic long tail distribution where many of the words are almost never used (and therefore much harder to learn good representations for.)\n\nThe other aspect is it treats related words completely separately. The set run/runs/running/ran are independent in this context, and a model needs to learn essentially the same concept multiple times.\n\nThis makes it clear why the convention is sub-word tokenisation. By defining a vocabulary set of character sequences we can balance flexibility and efficiency with the natural information compression that written languages have developed over thousands of years.\n\nThe principle is that common words can be taken whole (`the`\n\n= 1 token) and rare words can be broken into smaller reusable pieces (`tokenisation`\n\n→ `token`\n\n+ `isation`\n\n), that can be shared across words.\nIt also means we never actually find something inexpressible as even in the worst case you can break down new words into their component characters.\nThe question is just how to decide what the tokens are?\n\nThe dominant method (other tokenisers are available) is Byte Pair Encoding (BPE), which was invented in the 90s as a compression algorithm, and was adapted to the tokenisation problem in 2016.\n\nAt its heart this is a simple greedy algorithm. The procedure is as follows.\n\nYou start with a vocabulary of the individual chars, and a corpus of text you are using to compute your tokens from.\n\nThen you count adjacent pair frequencies across the corpus and merge the most frequent pair into a new token and add to vocab. Then repeat with the new vocabulary including the new token until you reach your target vocab size. We can write this more formally as:\n\nThen to apply the encoded vocabulary we replay the merge list in learned order. This contains a subtlety because the tokenisation is not just a lookup. The order is inherent to the tokeniser, as the order of the merges reflects the way the information has been compressed in the text.\n\nWhile BPE is dominant, it’s not the only scheme. Most variants change the way pairs are merged.\n\nWordPiece (as used in BERT) merges the pair that most increases the training corpus likelihood not just counting the raw frequency. This favours pairs that are more common than you would predict just from pair frequency.\nUnigram LM goes in the opposite direction. They start with a large candidate vocab, and then prune the tokens costing the least as measured by the likelihood. It keeps multiple segmentations, and samples among them in training (a form of mild data augmentation as a nice side effect).\nSentencePiece wraps the BPE/Unigram and treats space as an ordinary char (`▁`\n\n). This means they actually have a lossless detokenisation algorithm, and do not require language-specific whitespace splitting. This was used in Llama 1/2, T5, Gemma.\n\nWhatever the scheme the interaction with the core model is the same, the tokenised text acts as a lookup on an embedding table where each token has a learnt embedding vector.\n\nThere is a detail we have neglected here, we’ve talked about merging characters, but the question is, which characters? If we mean Unicode that’s >100k, and if we include all it’s going to bloat the base vocab, but excluding some will mean we have unknown tokens again. The fix introduced in GPT-2 (and now generally a standard) is to run BPE over raw bytes, not characters.\n\nEvery string (in any language or even emoji) can be represented as its UTF-8 byte sequence, which has only 256 byte values.\nWith 256 as the base vocab we can merge up and represent any string (again the worst case just requires individual bytes, which are all present), and no `[unk]`\n\never needed.\nIn GPT-2 the vocab size was 50,257.\n\nTokenisation is quite a complicated choice, and is often forgotten about when it comes to interpreting the model outputs, but it deeply impacts the way the model interacts with text, and many of the weirder failure modes as well.\n\nMost of these might feel broadly solved with the current generation of models, but usually require the use of a full reasoning thread to circumvent problems that are both intuitive to us, and feel much less complex than the kind of abilities most of these models have in other areas. Here are some of our favourites / classics of the genre:\n\nThe reason the counting explanation is shallow is that in reality there are two problems stacked together:\n\n(i) tokenisation hides the letter boundaries.\n\n(ii) counting is a serial operation that transformers do poorly in one forward pass, *even when* units are visible.\n\nAnd in a world pre chain of thought there isn’t a terribly natural way to “solve” this from a model perspective. Initially a straightforward Q/A response requires the next sentence that’s predicted to decompose, spell, and count in the internal representation of the model. Which generally does not work. And if you train your own GPT model from scratch, odds are that’s what it will do as well.\n\nThe model can be prompted to spell a token out letter by letter, and recover the individual spelling which gets around the problem in (i) partially. But the real issue here is a missing incentive, pretraining rarely encourages the model to decompose tokens back to chars.\n\nGiven the proper decomposition the task is easy, but the lengths required for such a simple task indicate how brittle this can be.\n\nThis is an even cleaner case, the longstanding difficulties of LLMs at basic arithmetic. The issue with addition and other basic arithmetic is that the meaning of the number is contained by both the position and the value (so far so similar to normal words) but also that each number is uniquely decomposable.\n\n`327`\n\n= 1 token; `1234`\n\n= `123`\n\n+ `4`\n\n; `12345`\n\n= `123`\n\n+ `45`\n\n, no exposed columns, and no consistent relationship between a number's value and its tokenisation.\nThe fix is direct, all we need to do is tokenise each digit separately or at least in a consistent way. Doing this, arithmetic accuracy improves significantly on an otherwise unchanged architecture.\n\nThis is something of a weird historical artefact, but in the early models people found obscure tokens that seemed to have strange properties and resulted in unexpected behaviour.\n\nThe classic example is `SolidGoldMagikarp`\n\n. This is a Reddit username that appeared frequently enough in the tokeniser's training data to get a vocab slot, but was filtered out of the model's training text, meaning the token never appeared.\nThe model was left with an embedding that was effectively untrained, almost at the initialisation state. So if you prompted it you’d get a near-random vector from an uninterpreted region which led to erratic output (sometimes evasion, often non-sequiturs, and occasional hostility).\n\nIt was found by searching for embeddings clustered near the centre of the distribution, i.e. ones which hadn’t moved from initialisation. This is a real example of the danger we mention above where the model corpus diverges from the tokeniser corpus. Fix is mundane, you need to keep the same training data for the two as much as possible, and prune these weird super rare tokens. Modern models don't generally have it, but it’s an engineering solution not a fundamental improvement.\n\nFor problems like this, whether it gets applied or not is a different question. Is a lab willing to spend the cost of retraining and revalidating the model and tokeniser together, or is it cheaper to just prune the offending tokens from the vocab and move on? The answer is usually the latter, even though it’s a brittle solution. And it doesn’t take much divergence between the two corpora to reintroduce the problem.\n\nThis is a subtler example, but is indicative of the complexity we have when dealing with quantised text. The boundary between the prompt and the continuation of the text.\nAt the byte-level tokenisers attach a leading space to the word i.e. `the`\n\nand ` the`\n\nare distinct tokens and therefore embeddings.\n\nHowever, when we have a trailing space in the prompt it means the next word is tokenised without the usual leading space. This leads to a worse-tokenised path than the same prompt would achieve without the space. It’s one of those weird effects that are hard to predict without the empirical evidence, and while the effect is small it is there. Like many of the issues discussed, once they are known they can be patched. So here, before generating you back up over the last token or two and re-tokenise across the boundary so the continuation splits the way training data would have. Not a terribly complex solution, but definitely bespoke.\n\nThe issue is with things like this is you don’t know what they are or how impactful they might be until they are discovered. While many have been identified, it is certain that there are many other effects that remain unpatched.\n\nThis is probably the most important failure mode, coming from an inequality in the way the data has been collected and the tokenisers trained. Fixing it is a matter of deliberate data collection and curation, which is much more onerous to perform well than the other fixes.\n\nEnglish is ≈ 1 byte/char, which is efficient, whereas a Chinese character usually requires 3 bytes, and emojis can take up to 4. When these are combined from the byte level it results in more tokens per unit of meaning. At the sentence level we might have 10 tokens for an English sentence, which can be inflated up to 14× in a non-Latin language. Given the fixed context size of models this means the information density is reduced and the model cost increased due to needing to parse longer sentences.\n\nTo see how this has changed over time we can look at how many tokens are required to quantise the same passage of text in different languages. Here we've used the Universal Declaration of Human Rights, which exists as a complete parallel translation in each language. For simplicity we count tokens relative to English on the same document. We found that the median token inflation across seventeen languages fell from 3.19× under GPT-2 (2019) to 1.45× under o200k (2024). And that for the largest non-Latin languages the gap remaining is shrinking. With the o200k tokeniser we see Arabic at 1.15×, Chinese at 1.18×, Russian at 1.35×, and Hindi at 1.57×.\n\nThe way this ~2× reduction was achieved was by increasing the vocabulary size, they went from roughly 50k to around 200k entries, mostly with a less English-weighted tokeniser training set. Crucially though, the merge algorithm didn't change.\n\nThe shift from GPT-2 to cl100k added about 50k slots that were 91.8% ASCII. Of the ASCII additions, 39% contain code punctuation such as `;`\n\nfollowed by a newline, or an opening brace and a newline. That expansion essentially went to English and to code.\nBy contrast when they went from cl100k to o200k they added about 100k slots and 47.1% were non-Latin script, and only 2% of the ASCII additions carry code punctuation. That expansion went to extending the multi-lingual coverage.\nAcross the whole vocabulary the non-Latin share went from 1.2% in GPT-2, to 3.0% in cl100k and finally up to 28.9% in o200k.\nThe jump comes at exactly the same time as the drop in token cost.\n\nThe convergence is not uniform across tokenisers of the same generation. DeepSeek V3 is the only tokeniser we measured where Chinese is cheaper than English (0.85×), and similarly Qwen3 gets Vietnamese to 1.41×, which is the lowest measured for that language. Hindi is still 4.88× there, among the highest, which probably reflects the intended userbase. In Llama 3.1 they extended cl100k by roughly 28k tokens. There English remained at 2206 tokens, while Russian fell from 5376 to 3462 and Hindi from 11444 to 6130, while Burmese is identical at 31028.\n\nLanguages that don't seem to be a commercial priority still have not converged. Burmese ranges from 3.09× (Gemma 3) to 14.07× (Llama 3.1), which will be a significant cost to any Burmese user, and is a clear example of the inequity that can arise from the tokenisation process.\n\nThe easy answer is to expand the coverage of all languages until parity is reached, but larger vocabularies are not free. Moving from 128k to 262k entries approximately doubles the embedding matrix, on the order of 0.5B parameters at a model dimension of 4096.\n\nThe risk of doing that badly is the one we already saw with `SolidGoldMagikarp`\n\n. Since a token whose training data is thin ends up sitting near its initialisation, expanding coverage faster than the data can support brings that failure back. This is a large part of why the answer isn’t simply to keep adding entries until parity.\n\nSo there is a tradeoff between improving the representation at the tokeniser level and the capacity the model has to learn good relationships for the tokens you add. Under-represented languages don’t come out of that tradeoff particularly well unless somebody targets them on purpose at the training stage, but unfortunately collecting that data is the hard part. The same document costs 31k tokens in Burmese against 2k in English, which is the gap that better coverage would close.\n\nThere’s a common argument posed to tokenisers, that they impose an unnecessary inductive bias onto the sequence and we should do away with them. All the failures we discuss here share a common pattern. They are all symptoms of the limitation imposed by having a fixed subword vocab, and we find fixes for all by adjusting the tokeniser/data/inference. So can we just remove the tokeniser entirely?\n\nThis has been attempted in various forms, all acting on the raw bytes of the text sequence. But while the performance is close, none have genuinely displaced BPE.\n\nThere are few fundamental changes that occur in LLM research, the exact nature of the attention vs state space vs diffusion models is interesting, but secondary to the building blocks of the objective function and how the data is quantised. If there is a change coming, this is where we would be watching.\n\nTokenisers are still fixed before any training example, and this means we trade vocab size against the sequence length. They are best seen as a pragmatic choice that makes for a more efficient model, but not a fundamental constraint.\n\nIt might be a pragmatic choice, but it isn't a neutral one. Arabic and Chinese now sit within 20% of English because there was a commercial reason to bring their cost down, while Burmese still costs 31k tokens against 2k for the same document because there wasn't. That discrepancy isn't a property of byte pair encoding, it just says something about who they're building the models for.", "url": "https://wpnews.pro/news/tokenisation-who-decides-what-a-token-is-anyway", "canonical_source": "https://idlemachines.co.uk/essays/tokenisation", "published_at": "2026-08-10 22:05:46+00:00", "updated_at": "2026-08-10 22:41:26.391696+00:00", "lang": "en", "topics": ["natural-language-processing", "large-language-models"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/tokenisation-who-decides-what-a-token-is-anyway", "markdown": "https://wpnews.pro/news/tokenisation-who-decides-what-a-token-is-anyway.md", "text": "https://wpnews.pro/news/tokenisation-who-decides-what-a-token-is-anyway.txt", "jsonld": "https://wpnews.pro/news/tokenisation-who-decides-what-a-token-is-anyway.jsonld"}}