BPE-Style Tokenizers: The Small Algorithm That Decides What an LLM Can See

Shrijith Venkatramana, an engineer building the AI code review tool LiveReview, traces modern LLM tokenization back to Byte Pair Encoding, a 1994 data-compression algorithm by Philip Gage. The writeup explains how BPE-style subword tokenization resolves the open-vocabulary problem by balancing short sequences against vocabulary size, and notes its 2016 adoption for neural machine translation by Rico Sennrich, Barry Haddow and Alexandra Birch before it became standard in models like GPT-2.

Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us https://github.com/HexmosTech/LiveReview/ to help devs discover the project, give it a try, and share your feedback to help improve the product. When you type: unbelievableness an LLM does not see the word. It sees something more like: "un", "believ", "ableness" Or perhaps: "un", "believe", "ness" Or, depending on the tokenizer: "un", "bel", "iev", "ab", "leness" That difference is not cosmetic. Tokenization determines the length of the model's input sequence, which affects context usage, inference cost, attention computation, vocabulary size, handling of rare words, programming-language behavior, multilingual performance, and even some model failure modes. And one of the most widely used ideas behind modern LLM tokenizers has an unusually non-LLM origin: a 1994 data-compression algorithm by a programmer named Philip Gage. The basic idea is remarkably simple: Find things that occur together often, and give them a reusable symbol. That idea eventually went from C programmers doing data compression, to neural machine translation, to GPT-2 and the tokenization machinery surrounding today's language models. This article builds the idea from intuition to implementation, then looks at the less obvious engineering consequences. A neural network wants numbers. Your input is text: The server returned HTTP 500. The model needs: the, server, returned, HTTP, 500, . which eventually becomes integer IDs such as: 464, 2126, 4710, ... The obvious question is: Why not make every word a token? Suppose the vocabulary contains: cat dog server database running ... Now consider: microarchitectural microarchitectures microarchitecturally You immediately run into the open-vocabulary problem. There are infinitely many possible strings. New product names appear. Developers invent identifiers. People misspell things. Languages generate long compounds. Users paste URLs, hashes, code, emojis and arbitrary Unicode. A word-level tokenizer therefore needs some fallback mechanism. At the other extreme, we could tokenize one character at a time: m i c r o a r c h i t e c t u r a l Now everything is representable, but sequences become much longer. That creates a fundamental tradeoff: word tokens <- shorter sequences, huge vocabulary, poor handling of unknown words character tokens <- tiny vocabulary, very long sequences subword tokens <- compromise BPE-style tokenization lives in that middle ground. Frequent sequences become single tokens. Rare sequences remain decomposable into smaller units. That is the key intuition. In 1994, Philip Gage published an article in The C Users Journal describing Byte Pair Encoding , or BPE. His original problem had nothing to do with language models. The idea was ordinary compression: Suppose data contains: ABABABABABAB and AB occurs constantly. Instead of repeatedly storing: A B A B A B A B ... we can create a new symbol representing: AB and replace occurrences of the pair. Do it repeatedly, and common sequences become increasingly compact. The original algorithm therefore looked roughly like: find the most frequent adjacent byte pair replace it with a new symbol repeat This is a compression algorithm. But the basic mechanism turns out to be useful for language. In 2016, Rico Sennrich, Barry Haddow and Alexandra Birch applied BPE to neural machine translation. Their motivation was the open-vocabulary problem : machine translation systems had to deal with names, compounds and rare words that could not reasonably all appear in a fixed word vocabulary. Consider: counterrevolutionaries A word-level vocabulary might not contain it. A subword system could represent it approximately as: counter + revolution + ar + ies The exact segmentation is learned from data rather than being supplied by a linguist. This mattered because the model could now encounter a word it had never seen as a whole while still having a representation for its pieces. Then GPT-2 made an important variation mainstream: byte-level BPE . Instead of starting from all Unicode characters, GPT-2 starts from the 256 possible byte values. That gives a tiny guaranteed base vocabulary while preserving the ability to represent arbitrary byte sequences. GPT-2 used a vocabulary of 50,257 entries, consisting of the 256-byte base plus 50,000 learned merges and a special token. OpenAI CDN https://cdn.openai.com/better-language-models/language-models.pdf So the lineage is roughly: 1994: byte compression | v 2016: subword representation for NMT | v 2019: byte-level BPE for GPT-2 | v modern LLM tokenizers The interesting part is that almost none of this requires a sophisticated linguistic theory. It is mostly frequency statistics plus a greedy merging procedure. Let's construct a tiny tokenizer. Suppose our corpus is: low low low low low lower lower widest widest widest newest newest newest newest newest newest First, pretend our base vocabulary consists of individual characters. We represent: low as: l o w and: lower l o w e r Now count adjacent pairs. For example: l, o o, w w, e e, r w, i i, d d, e e, s s, t n, e Because newest appears six times, the pair: e, s appears six times. Likewise: s, t BPE asks: Which adjacent pair is most frequent? Suppose we pick: e, s and create a new symbol: es Now: newest becomes: n e w es t The vocabulary has grown by one. Next we recount pairs and may discover: es, t is highly frequent. Merge again: est newest n e w est Continue. Eventually you might learn: st est west newest depending on corpus frequencies and the exact sequence of merges. The algorithm is therefore almost embarrassingly simple. Let the current token sequence for a corpus be made from symbols in vocabulary V . For every adjacent pair a, b , compute its frequency: f a, b = number of times a is immediately followed by b Then choose: a , b = argmax a,b f a, b Create a new token: c = a || b where || means concatenation. Then replace every occurrence of: a b with: c and repeat. If we begin with B base symbols and perform K merges: |V| = B + K + special tokens For byte-level BPE: B = 256 So with 50,000 merges: |V| ~= 50,000 + 256 plus whatever special tokens the system uses. This is a useful mental model: The tokenizer vocabulary is largely a compressed dictionary of frequently useful byte sequences. There is an important property hiding inside the greedy algorithm. Suppose these sequences are common: tion ing pre un http :// BPE will tend to discover them because they occur frequently. Eventually it may discover larger units: communicat + ion or perhaps: commun + ication or, for a very common word: communication as one complete token. This means the tokenizer automatically creates something resembling a hierarchy: php bytes - small fragments - common morpheme-like units - common words - common multi-character sequences But an important distinction: BPE does not understand morphology. It does not know that: walk walking walked walker share a linguistic stem. It only knows that certain byte sequences occur frequently enough to be worth merging. That distinction matters when people say things like "the tokenizer understands prefixes." It does not. It has learned a segmentation that is useful according to its training statistics. Imagine a corpus where: hyperparameter occurs 50,000 times. Then the tokenizer has an economic incentive, in vocabulary terms, to represent something like: hyperparameter compactly. But suppose: hyperparametrix appears once. A BPE tokenizer can still represent it: hyper + parameter + ix or some other decomposition. This is the main advantage over word-level tokenization. It gets compression for common patterns without making the vocabulary responsible for every possible word . <unk Ordinary character-level BPE has an awkward problem. Unicode is enormous. If you want every possible Unicode character to be a base symbol, your initial vocabulary is already huge. GPT-2 instead starts from bytes. There are exactly: 256 possible byte values. Any Unicode string encoded as UTF-8 becomes a byte sequence: php text - UTF-8 - bytes - BPE merges - token IDs This has an important consequence: there is always a fallback representation. Even if a tokenizer has never seen a particular Unicode string during training, the raw bytes can still be represented. For example, an emoji such as: 👍 is represented internally by its UTF-8 bytes: F0 9F 91 8D The tokenizer may have learned to merge those bytes, partially merge them, or leave them separate. But it does not need a vocabulary entry literally corresponding to every possible Unicode character. That is a powerful design decision. Naively running BPE over raw bytes has undesirable behavior. Suppose your corpus contains: dog dog. dog dog? Frequency-based BPE may learn variants of entire sequences that are statistically frequent, wasting vocabulary entries on punctuation-specific combinations. GPT-2's approach therefore constrained which byte sequences could merge, while treating spaces specially. The objective was to retain the generality of byte-level representation without allowing the greedy learner to spend too much vocabulary capacity on accidental boundary variants. OpenAI CDN https://cdn.openai.com/better-language-models/language-models.pdf This is a recurring theme in tokenizer engineering: The basic algorithm is simple. Most of the engineering is deciding where the simple algorithm is allowed to operate. This is where tokenization stops being an NLP curiosity. Consider a model with a context window of: 128,000 tokens If your tokenizer turns a piece of text into: 100,000 tokens you have room for approximately: 28,000 tokens of additional context. If another tokenizer represents exactly the same text as: 80,000 tokens you now have approximately: 48,000 tokens left. That is a 71% increase in remaining context. The difference gets even more important for long-context workloads. For standard full self-attention, the interaction matrix is approximately: n x n so the dominant attention computation scales approximately as: O n^2 Suppose tokenizer A gives you: n = 10,000 tokens. Tokenizer B produces 20% more: n = 12,000 The ratio of pairwise attention work is approximately: 12,000^2 / 10,000^2 = 1.44 So a 20% increase in token count can imply roughly: 44% more pairwise attention work. That is not a property of BPE itself. It is a consequence of the fact that tokenization controls sequence length . This gives us a useful engineering principle: php characters - tokenizer - token count - context utilization - compute + memory + latency Imagine two representations of the same sentence: Tokenizer A: 12 tokens Tokenizer B: 18 tokens The model using B has to predict a longer sequence. At training time that means more prediction positions. At inference time it means more autoregressive steps. For APIs, token count also becomes a billing and capacity unit because providers commonly meter usage in tokens. So tokenizer quality is not merely: "Does the text tokenize?" It is also: "How economically does this representation use the model's finite sequence budget?" python def calculate monthly revenue customer transactions : ... A tokenizer that is optimized around English prose may discover useful units such as: calculate monthly revenue customer But source code contains many patterns that have different frequency distributions: js init HTTPRequest std::unordered map get user profile === Programming languages are therefore an interesting tokenizer workload because identifiers, punctuation, whitespace, delimiters and repeated syntactic fragments all compete for vocabulary capacity. The result is one reason why "tokenizer efficiency" should be evaluated on the actual distribution your model serves, not only on generic English text. Once training is finished, the tokenizer no longer needs to "discover" anything. It has two important artifacts: vocabulary merge rules For example, imagine the merge ranking contains: 1. e s 2. es t 3. n e 4. ne w 5. new est ... Now given: newest the encoder applies the learned rules in their defined priority. Conceptually: n e w e s t then perhaps: ne w e s t then: ne w est then eventually: new est depending on the learned merge table. The output is something like: new, est The exact implementation used by modern tokenizers is optimized considerably beyond this toy procedure. A naive implementation that rescans an entire corpus after every merge would be unnecessarily expensive. But the conceptual model remains: base symbols + ordered merge rules = tokenizer And that has a subtle consequence for developers: token IDs are meaningless without the tokenizer definition that produced them. Token ID: 12345 does not inherently mean "hello" or "database." It means whatever entry 12345 refers to in a particular tokenizer vocabulary. This is also why changing tokenizers can invalidate embeddings, model inputs, cached token sequences and various pieces of preprocessing infrastructure. The tokenizer is effectively part of the model's interface contract. BPE solves one problem very well: How do we turn arbitrary text into a finite vocabulary while giving common sequences compact representations? It does not solve everything. It does not guarantee linguistically meaningful boundaries. It does not guarantee equal token efficiency across languages. It does not make arithmetic easy. It does not make code identifiers naturally interpretable. It does not prevent pathological tokenizations. And it certainly does not give the model a semantic understanding of the pieces. You can see this clearly with a made-up identifier: calculateUserMonthlyNetRevenueExcludingRefunds The tokenizer might produce something like: calculate User Monthly Net Revenue Excluding Refund s Or something considerably less intuitive. That is perfectly fine from the tokenizer's perspective. Its job is not to discover what the identifier "means." Its job is to produce a sequence that fits within the vocabulary and represents the input efficiently according to patterns learned from its corpus. This also explains an important phenomenon when working with LLM APIs: two strings that humans consider almost identical can have materially different token counts. camelCaseIdentifier snake case identifier may produce different segmentations because their character sequences and punctuation patterns have different statistics. hello world hello world are linguistically related but are not equivalent objects to a frequency-based tokenizer. The model ultimately sees the tokens, not our intuitive notion of "the same phrase." There is a useful way to think about the whole system. Your original text contains enormous redundancy. BPE performs a kind of learned compression: raw bytes | v frequent local patterns | v reusable subword tokens | v shorter sequence | v Transformer The irony is that the algorithm is not particularly sophisticated. Count adjacent pairs. Merge the frequent ones. Repeat. Yet that small mechanism sits directly in front of billions of neural-network parameters. And its decisions propagate everywhere: php tokenizer - sequence length - context capacity - attention computation - inference latency - memory usage - training efficiency - API cost - multilingual behavior - code handling That makes tokenization one of those pieces of infrastructure that is easy to ignore precisely because it works so well. The most interesting lesson may be historical. Philip Gage was trying to compress bytes in 1994. Sennrich, Haddow and Birch were trying to solve rare-word problems in neural translation in 2016. GPT-2 then adapted the idea to byte-level language modeling. A concept that began as a compact data-compression trick became part of the interface between human language and modern neural networks. That is a useful reminder for developers building ML systems: sometimes the important abstraction is not the complicated algorithm in the middle, but the small transformation that determines what the algorithm gets to see. What tokenization behavior have you found most counterintuitive in an LLM—code, multilingual text, numbers, punctuation, or something else? Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down. I'm building LiveReview , a blast-radius aware AI code review built for your business-critical systems. Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters. Spend code review effort where business risk is highest — not spread evenly across every diff. ⭐ Star it on GitHub: LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius : how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff. LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer. | The exact math, not a black box | Visualize blast radius at a glance | Every factor that feeds the score | |---|---|---| Here's the goal: Click below to try LiveReview with your codebase: