{"slug": "bpe-style-tokenizers-the-small-algorithm-that-decides-what-an-llm-can-see", "title": "BPE-Style Tokenizers: The Small Algorithm That Decides What an LLM Can See", "summary": "Shrijith Venkatramana, an engineer building the AI code review tool LiveReview, traces modern LLM tokenization back to Byte Pair Encoding, a 1994 data-compression algorithm by Philip Gage. The writeup explains how BPE-style subword tokenization resolves the open-vocabulary problem by balancing short sequences against vocabulary size, and notes its 2016 adoption for neural machine translation by Rico Sennrich, Barry Haddow and Alexandra Birch before it became standard in models like GPT-2.", "body_md": "*Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. [Star us](https://github.com/HexmosTech/LiveReview/) to help devs discover the project, give it a try, and share your feedback to help improve the product.*\n\nWhen you type:\n\n```\nunbelievableness\n```\n\nan LLM does not see the word.\n\nIt sees something more like:\n\n```\n[\"un\", \"believ\", \"ableness\"]\n```\n\nOr perhaps:\n\n```\n[\"un\", \"believe\", \"ness\"]\n```\n\nOr, depending on the tokenizer:\n\n```\n[\"un\", \"bel\", \"iev\", \"ab\", \"leness\"]\n```\n\nThat difference is not cosmetic.\n\nTokenization determines the length of the model's input sequence, which affects context usage, inference cost, attention computation, vocabulary size, handling of rare words, programming-language behavior, multilingual performance, and even some model failure modes.\n\nAnd one of the most widely used ideas behind modern LLM tokenizers has an unusually non-LLM origin:\n\n**a 1994 data-compression algorithm by a programmer named Philip Gage.**\n\nThe basic idea is remarkably simple:\n\nFind things that occur together often, and give them a reusable symbol.\n\nThat idea eventually went from C programmers doing data compression, to neural machine translation, to GPT-2 and the tokenization machinery surrounding today's language models.\n\nThis article builds the idea from intuition to implementation, then looks at the less obvious engineering consequences.\n\nA neural network wants numbers.\n\nYour input is text:\n\n```\nThe server returned HTTP 500.\n```\n\nThe model needs:\n\n```\n[the, server, returned, HTTP, 500, .]\n```\n\nwhich eventually becomes integer IDs such as:\n\n```\n[464, 2126, 4710, ...]\n```\n\nThe obvious question is:\n\n**Why not make every word a token?**\n\nSuppose the vocabulary contains:\n\n```\ncat\ndog\nserver\ndatabase\nrunning\n...\n```\n\nNow consider:\n\n```\nmicroarchitectural\nmicroarchitectures\nmicroarchitecturally\n```\n\nYou immediately run into the open-vocabulary problem.\n\nThere are infinitely many possible strings. New product names appear. Developers invent identifiers. People misspell things. Languages generate long compounds. Users paste URLs, hashes, code, emojis and arbitrary Unicode.\n\nA word-level tokenizer therefore needs some fallback mechanism.\n\nAt the other extreme, we could tokenize one character at a time:\n\n```\nm i c r o a r c h i t e c t u r a l\n```\n\nNow everything is representable, but sequences become much longer.\n\nThat creates a fundamental tradeoff:\n\n```\nword tokens       <- shorter sequences, huge vocabulary, poor handling of unknown words\ncharacter tokens  <- tiny vocabulary, very long sequences\nsubword tokens    <- compromise\n```\n\nBPE-style tokenization lives in that middle ground.\n\nFrequent sequences become single tokens.\n\nRare sequences remain decomposable into smaller units.\n\nThat is the key intuition.\n\nIn 1994, Philip Gage published an article in *The C Users Journal* describing **Byte Pair Encoding**, or BPE.\n\nHis original problem had nothing to do with language models.\n\nThe idea was ordinary compression:\n\nSuppose data contains:\n\n```\nABABABABABAB\n```\n\nand `AB` occurs constantly.\n\nInstead of repeatedly storing:\n\n```\nA B A B A B A B ...\n```\n\nwe can create a new symbol representing:\n\n```\nAB\n```\n\nand replace occurrences of the pair.\n\nDo it repeatedly, and common sequences become increasingly compact.\n\nThe original algorithm therefore looked roughly like:\n\n```\nfind the most frequent adjacent byte pair\nreplace it with a new symbol\nrepeat\n```\n\nThis is a compression algorithm.\n\nBut the basic mechanism turns out to be useful for language.\n\nIn 2016, Rico Sennrich, Barry Haddow and Alexandra Birch applied BPE to neural machine translation. Their motivation was the **open-vocabulary problem**: machine translation systems had to deal with names, compounds and rare words that could not reasonably all appear in a fixed word vocabulary.\n\nConsider:\n\n```\ncounterrevolutionaries\n```\n\nA word-level vocabulary might not contain it.\n\nA subword system could represent it approximately as:\n\n```\ncounter + revolution + ar + ies\n```\n\nThe exact segmentation is learned from data rather than being supplied by a linguist.\n\nThis mattered because the model could now encounter a word it had never seen as a whole while still having a representation for its pieces.\n\nThen GPT-2 made an important variation mainstream: **byte-level BPE**.\n\nInstead of starting from all Unicode characters, GPT-2 starts from the 256 possible byte values. That gives a tiny guaranteed base vocabulary while preserving the ability to represent arbitrary byte sequences. GPT-2 used a vocabulary of 50,257 entries, consisting of the 256-byte base plus 50,000 learned merges and a special token. ([OpenAI CDN](https://cdn.openai.com/better-language-models/language-models.pdf))\n\nSo the lineage is roughly:\n\n```\n1994: byte compression\n        |\n        v\n2016: subword representation for NMT\n        |\n        v\n2019: byte-level BPE for GPT-2\n        |\n        v\nmodern LLM tokenizers\n```\n\nThe interesting part is that almost none of this requires a sophisticated linguistic theory.\n\nIt is mostly frequency statistics plus a greedy merging procedure.\n\nLet's construct a tiny tokenizer.\n\nSuppose our corpus is:\n\n```\nlow low low low low\nlower lower\nwidest widest widest\nnewest newest newest newest newest newest\n```\n\nFirst, pretend our base vocabulary consists of individual characters.\n\nWe represent:\n\n```\nlow\n```\n\nas:\n\n```\nl o w\n```\n\nand:\n\n```\nlower\nl o w e r\n```\n\nNow count adjacent pairs.\n\nFor example:\n\n```\n(l, o)\n(o, w)\n(w, e)\n(e, r)\n(w, i)\n(i, d)\n(d, e)\n(e, s)\n(s, t)\n(n, e)\n```\n\nBecause `newest` appears six times, the pair:\n\n```\n(e, s)\n```\n\nappears six times.\n\nLikewise:\n\n```\n(s, t)\n```\n\nBPE asks:\n\n```\nWhich adjacent pair is most frequent?\n```\n\nSuppose we pick:\n\n```\n(e, s)\n```\n\nand create a new symbol:\n\n```\nes\n```\n\nNow:\n\n```\nnewest\n```\n\nbecomes:\n\n```\nn e w es t\n```\n\nThe vocabulary has grown by one.\n\nNext we recount pairs and may discover:\n\n```\n(es, t)\n```\n\nis highly frequent.\n\nMerge again:\n\n```\nest\nnewest\nn e w est\n```\n\nContinue.\n\nEventually you might learn:\n\n```\nst\nest\nwest\nnewest\n```\n\ndepending on corpus frequencies and the exact sequence of merges.\n\nThe algorithm is therefore almost embarrassingly simple.\n\nLet the current token sequence for a corpus be made from symbols in vocabulary `V`.\n\nFor every adjacent pair `(a, b)`, compute its frequency:\n\n```\nf(a, b) = number of times a is immediately followed by b\n```\n\nThen choose:\n\n```\n(a*, b*) = argmax_(a,b) f(a, b)\n```\n\nCreate a new token:\n\n```\nc = a || b\n```\n\nwhere `||` means concatenation.\n\nThen replace every occurrence of:\n\n```\na b\n```\n\nwith:\n\n```\nc\n```\n\nand repeat.\n\nIf we begin with `B` base symbols and perform `K` merges:\n\n```\n|V| = B + K + special_tokens\n```\n\nFor byte-level BPE:\n\n```\nB = 256\n```\n\nSo with 50,000 merges:\n\n```\n|V| ~= 50,000 + 256\n```\n\nplus whatever special tokens the system uses.\n\nThis is a useful mental model:\n\n**The tokenizer vocabulary is largely a compressed dictionary of frequently useful byte sequences.**\n\nThere is an important property hiding inside the greedy algorithm.\n\nSuppose these sequences are common:\n\n```\ntion\ning\npre\nun\nhttp\n://\n```\n\nBPE will tend to discover them because they occur frequently.\n\nEventually it may discover larger units:\n\n```\ncommunicat + ion\n```\n\nor perhaps:\n\n```\ncommun + ication\n```\n\nor, for a very common word:\n\n```\ncommunication\n```\n\nas one complete token.\n\nThis means the tokenizer automatically creates something resembling a hierarchy:\n\n``` php\nbytes\n  ->\nsmall fragments\n  ->\ncommon morpheme-like units\n  ->\ncommon words\n  ->\ncommon multi-character sequences\n```\n\nBut an important distinction:\n\n**BPE does not understand morphology.**\n\nIt does not know that:\n\n```\nwalk\nwalking\nwalked\nwalker\n```\n\nshare a linguistic stem.\n\nIt only knows that certain byte sequences occur frequently enough to be worth merging.\n\nThat distinction matters when people say things like \"the tokenizer understands prefixes.\"\n\nIt does not.\n\nIt has learned a segmentation that is useful according to its training statistics.\n\nImagine a corpus where:\n\n```\nhyperparameter\n```\n\noccurs 50,000 times.\n\nThen the tokenizer has an economic incentive, in vocabulary terms, to represent something like:\n\n```\nhyperparameter\n```\n\ncompactly.\n\nBut suppose:\n\n```\nhyperparametrix\n```\n\nappears once.\n\nA BPE tokenizer can still represent it:\n\n```\nhyper + parameter + ix\n```\n\nor some other decomposition.\n\nThis is the main advantage over word-level tokenization.\n\nIt gets **compression for common patterns without making the vocabulary responsible for every possible word**.\n\n`<unk>`\nOrdinary character-level BPE has an awkward problem.\n\nUnicode is enormous.\n\nIf you want every possible Unicode character to be a base symbol, your initial vocabulary is already huge.\n\nGPT-2 instead starts from bytes.\n\nThere are exactly:\n\n```\n256\n```\n\npossible byte values.\n\nAny Unicode string encoded as UTF-8 becomes a byte sequence:\n\n``` php\ntext\n  ->\nUTF-8\n  ->\nbytes\n  ->\nBPE merges\n  ->\ntoken IDs\n```\n\nThis has an important consequence:\n\n**there is always a fallback representation.**\n\nEven if a tokenizer has never seen a particular Unicode string during training, the raw bytes can still be represented.\n\nFor example, an emoji such as:\n\n```\n👍\n```\n\nis represented internally by its UTF-8 bytes:\n\n```\nF0 9F 91 8D\n```\n\nThe tokenizer may have learned to merge those bytes, partially merge them, or leave them separate.\n\nBut it does not need a vocabulary entry literally corresponding to every possible Unicode character.\n\nThat is a powerful design decision.\n\nNaively running BPE over raw bytes has undesirable behavior.\n\nSuppose your corpus contains:\n\n```\ndog\ndog.\ndog!\ndog?\n```\n\nFrequency-based BPE may learn variants of entire sequences that are statistically frequent, wasting vocabulary entries on punctuation-specific combinations.\n\nGPT-2's approach therefore constrained which byte sequences could merge, while treating spaces specially. The objective was to retain the generality of byte-level representation without allowing the greedy learner to spend too much vocabulary capacity on accidental boundary variants. ([OpenAI CDN](https://cdn.openai.com/better-language-models/language-models.pdf))\n\nThis is a recurring theme in tokenizer engineering:\n\n**The basic algorithm is simple. Most of the engineering is deciding where the simple algorithm is allowed to operate.**\n\nThis is where tokenization stops being an NLP curiosity.\n\nConsider a model with a context window of:\n\n```\n128,000 tokens\n```\n\nIf your tokenizer turns a piece of text into:\n\n```\n100,000 tokens\n```\n\nyou have room for approximately:\n\n```\n28,000 tokens\n```\n\nof additional context.\n\nIf another tokenizer represents exactly the same text as:\n\n```\n80,000 tokens\n```\n\nyou now have approximately:\n\n```\n48,000 tokens\n```\n\nleft.\n\nThat is a 71% increase in remaining context.\n\nThe difference gets even more important for long-context workloads.\n\nFor standard full self-attention, the interaction matrix is approximately:\n\n```\nn x n\n```\n\nso the dominant attention computation scales approximately as:\n\n```\nO(n^2)\n```\n\nSuppose tokenizer A gives you:\n\n```\nn = 10,000\n```\n\ntokens.\n\nTokenizer B produces 20% more:\n\n```\nn = 12,000\n```\n\nThe ratio of pairwise attention work is approximately:\n\n```\n12,000^2 / 10,000^2\n= 1.44\n```\n\nSo a 20% increase in token count can imply roughly:\n\n```\n44% more\n```\n\npairwise attention work.\n\nThat is not a property of BPE itself. It is a consequence of the fact that **tokenization controls sequence length**.\n\nThis gives us a useful engineering principle:\n\n``` php\ncharacters\n    ->\ntokenizer\n    ->\ntoken count\n    ->\ncontext utilization\n    ->\ncompute + memory + latency\n```\n\nImagine two representations of the same sentence:\n\n```\nTokenizer A: 12 tokens\nTokenizer B: 18 tokens\n```\n\nThe model using B has to predict a longer sequence.\n\nAt training time that means more prediction positions.\n\nAt inference time it means more autoregressive steps.\n\nFor APIs, token count also becomes a billing and capacity unit because providers commonly meter usage in tokens.\n\nSo tokenizer quality is not merely:\n\n```\n\"Does the text tokenize?\"\n```\n\nIt is also:\n\n```\n\"How economically does this representation use the model's finite sequence budget?\"\npython\ndef calculate_monthly_revenue(customer_transactions):\n    ...\n```\n\nA tokenizer that is optimized around English prose may discover useful units such as:\n\n```\ncalculate\nmonthly\nrevenue\ncustomer\n```\n\nBut source code contains many patterns that have different frequency distributions:\n\n``` js\n__init__\nHTTPRequest\nstd::unordered_map\nget_user_profile\n===>\n```\n\nProgramming languages are therefore an interesting tokenizer workload because identifiers, punctuation, whitespace, delimiters and repeated syntactic fragments all compete for vocabulary capacity.\n\nThe result is one reason why \"tokenizer efficiency\" should be evaluated on the actual distribution your model serves, not only on generic English text.\n\nOnce training is finished, the tokenizer no longer needs to \"discover\" anything.\n\nIt has two important artifacts:\n\n```\nvocabulary\nmerge rules\n```\n\nFor example, imagine the merge ranking contains:\n\n```\n1.  e s\n2.  es t\n3.  n e\n4.  ne w\n5.  new est\n...\n```\n\nNow given:\n\n```\nnewest\n```\n\nthe encoder applies the learned rules in their defined priority.\n\nConceptually:\n\n```\nn e w e s t\n```\n\nthen perhaps:\n\n```\nne w e s t\n```\n\nthen:\n\n```\nne w est\n```\n\nthen eventually:\n\n```\nnew est\n```\n\ndepending on the learned merge table.\n\nThe output is something like:\n\n```\n[new, est]\n```\n\nThe exact implementation used by modern tokenizers is optimized considerably beyond this toy procedure. A naive implementation that rescans an entire corpus after every merge would be unnecessarily expensive.\n\nBut the conceptual model remains:\n\n```\nbase symbols\n    +\nordered merge rules\n    =\ntokenizer\n```\n\nAnd that has a subtle consequence for developers:\n\n**token IDs are meaningless without the tokenizer definition that produced them.**\n\nToken ID:\n\n```\n12345\n```\n\ndoes not inherently mean \"hello\" or \"database.\"\n\nIt means whatever entry 12345 refers to in a particular tokenizer vocabulary.\n\nThis is also why changing tokenizers can invalidate embeddings, model inputs, cached token sequences and various pieces of preprocessing infrastructure.\n\nThe tokenizer is effectively part of the model's interface contract.\n\nBPE solves one problem very well:\n\nHow do we turn arbitrary text into a finite vocabulary while giving common sequences compact representations?\n\nIt does not solve everything.\n\nIt does not guarantee linguistically meaningful boundaries.\n\nIt does not guarantee equal token efficiency across languages.\n\nIt does not make arithmetic easy.\n\nIt does not make code identifiers naturally interpretable.\n\nIt does not prevent pathological tokenizations.\n\nAnd it certainly does not give the model a semantic understanding of the pieces.\n\nYou can see this clearly with a made-up identifier:\n\n```\ncalculateUserMonthlyNetRevenueExcludingRefunds\n```\n\nThe tokenizer might produce something like:\n\n```\ncalculate\nUser\nMonthly\nNet\nRevenue\nExcluding\nRefund\ns\n```\n\nOr something considerably less intuitive.\n\nThat is perfectly fine from the tokenizer's perspective.\n\nIts job is not to discover what the identifier \"means.\"\n\nIts job is to produce a sequence that fits within the vocabulary and represents the input efficiently according to patterns learned from its corpus.\n\nThis also explains an important phenomenon when working with LLM APIs:\n\n**two strings that humans consider almost identical can have materially different token counts.**\n\n```\ncamelCaseIdentifier\nsnake_case_identifier\n```\n\nmay produce different segmentations because their character sequences and punctuation patterns have different statistics.\n\n```\nhello world\nhello_world\n```\n\nare linguistically related but are not equivalent objects to a frequency-based tokenizer.\n\nThe model ultimately sees the tokens, not our intuitive notion of \"the same phrase.\"\n\nThere is a useful way to think about the whole system.\n\nYour original text contains enormous redundancy.\n\nBPE performs a kind of learned compression:\n\n```\nraw bytes\n   |\n   v\nfrequent local patterns\n   |\n   v\nreusable subword tokens\n   |\n   v\nshorter sequence\n   |\n   v\nTransformer\n```\n\nThe irony is that the algorithm is not particularly sophisticated.\n\nCount adjacent pairs.\n\nMerge the frequent ones.\n\nRepeat.\n\nYet that small mechanism sits directly in front of billions of neural-network parameters.\n\nAnd its decisions propagate everywhere:\n\n``` php\ntokenizer\n   -> sequence length\n   -> context capacity\n   -> attention computation\n   -> inference latency\n   -> memory usage\n   -> training efficiency\n   -> API cost\n   -> multilingual behavior\n   -> code handling\n```\n\nThat makes tokenization one of those pieces of infrastructure that is easy to ignore precisely because it works so well.\n\nThe most interesting lesson may be historical.\n\nPhilip Gage was trying to compress bytes in 1994. Sennrich, Haddow and Birch were trying to solve rare-word problems in neural translation in 2016. GPT-2 then adapted the idea to byte-level language modeling.\n\nA concept that began as a compact data-compression trick became part of the interface between human language and modern neural networks.\n\nThat is a useful reminder for developers building ML systems:\n\n**sometimes the important abstraction is not the complicated algorithm in the middle, but the small transformation that determines what the algorithm gets to see.**\n\nWhat tokenization behavior have you found most counterintuitive in an LLM—code, multilingual text, numbers, punctuation, or something else?\n\nYour team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.\n\nI'm building **LiveReview**, a blast-radius aware AI code review built for your business-critical systems.\n\nInstead of presenting every diff with equal emphasis, **LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.**\n\nSpend code review effort where business risk is highest — not spread evenly across every diff.\n\n⭐ Star it on GitHub: \n\nLiveReview is an AI code reviewer that scores every hunk of a diff by **blast radius**: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.\n\n*LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.*\n\n| The exact math, not a black box | Visualize blast radius at a glance | Every factor that feeds the score | \n|---|---|---|\n\n**Here's the goal:**\n\n**Click below to try LiveReview with your codebase:**", "url": "https://wpnews.pro/news/bpe-style-tokenizers-the-small-algorithm-that-decides-what-an-llm-can-see", "canonical_source": "https://dev.to/shrsv/bpe-style-tokenizers-the-small-algorithm-that-decides-what-an-llm-can-see-2a72", "published_at": "2026-09-12 19:11:21+00:00", "updated_at": "2026-09-12 19:23:51.380023+00:00", "lang": "en", "topics": ["natural-language-processing", "large-language-models", "machine-learning", "ai-research"], "entities": ["Shrijith Venkatramana", "LiveReview", "Philip Gage", "Rico Sennrich", "Barry Haddow", "Alexandra Birch", "GPT-2", "Byte Pair Encoding"], "alternates": {"html": "https://wpnews.pro/news/bpe-style-tokenizers-the-small-algorithm-that-decides-what-an-llm-can-see", "markdown": "https://wpnews.pro/news/bpe-style-tokenizers-the-small-algorithm-that-decides-what-an-llm-can-see.md", "text": "https://wpnews.pro/news/bpe-style-tokenizers-the-small-algorithm-that-decides-what-an-llm-can-see.txt", "jsonld": "https://wpnews.pro/news/bpe-style-tokenizers-the-small-algorithm-that-decides-what-an-llm-can-see.jsonld"}}