{"slug": "what-makes-llm-tokenization-slow", "title": "What Makes LLM Tokenization Slow?", "summary": "LLM tokenization, while a small part of overall latency, sits on the hot path and can occur multiple times per request, according to a technical analysis of GPT-2's reference encoder. The analysis shows that converting text to token IDs via byte-pair encoding (BPE) involves pre-tokenization with regex, merging byte pairs based on trained ranks, and looking up token strings in a vocabulary, with the example 'goldshire' requiring eight merge rounds to become two token IDs. The author ported the encoder to Rust to explore performance improvements.", "body_md": "LLMs receive sequences of token IDs instead of raw text. For example, the GPT-2 tokenizer encodes `what's the weather in goldshire?`\n\ninto `[10919, 338, 262, 6193, 287, 3869, 10932, 30]`\n\n.\n\nWhile it's a small part of the overall latency of a modern LLM call, tokenization does sit on the hot path. In some LLM products, it may even happen multiple times, e.g. to decide if it's time to compress the context, estimating cost, or how to route the prompt when it lands in the provider's infrastructure.\n\nI wanted to learn more about how tokenization works and what the performance constraints are. I chose to study GPT-2's [reference encoder](https://github.com/openai/gpt-2/blob/master/src/encoder.py) because it's compact and readable. Even though this post will be fairly GPT-2-specific, the underlying ideas and performance considerations haven't changed that much in the time since.\n\nAs you'll see in the following sections, I converted GPT-2's reference encoder to Rust and then tried to make it a lot faster.\n\n## Quick Intro to Byte-Pair Encoding\n\nGPT-2 uses [byte-pair encoding](https://en.wikipedia.org/wiki/Byte-pair_encoding), or BPE. It converts arbitrary UTF-8 text into a reversible sequence of token IDs. It starts from byte symbols and uses a trained list of merge ranks to make common sequences into single tokens.\n\nBefore this, GPT-2 runs a [regex](https://github.com/openai/gpt-2/blob/master/src/encoder.py#L53C20-L53C113) over the input to divide it into regions that BPE will process separately (word-like text, numbers, contractions, punctuation, and whitespace).\n\n```\nwhat's the weather in goldshire?             |             v[what] ['s] [ the] [ weather] [ in] [ goldshire] [?]\n```\n\nThis pre-tokenization step stops BPE from merging across obvious changes in text type. For example, `goldshire`\n\nand `?`\n\nare separate regions, so BPE should not try to merge across that boundary. The goal of the regex is to provide some weak assumptions about where useful boundaries probably exist.\n\nGPT-2 needs all 256 byte values as its base alphabet in order to represent arbitrary UTF-8 without an unknown-token fallback. Literal bytes include spaces, controls, and invalid standalone UTF-8 values, so `encoder.py`\n\nmaps them to safe visible Unicode symbols for string-based BPE, e.g. UTF-8 `é`\n\nis bytes `c3 a9`\n\n, represented as `é`\n\n.\n\nIn the GPT-2 source, `vocab.bpe`\n\nlists the pairs that may merge, with earlier lines having higher priority. I've trimmed it to show all the pairs needed to merge `goldshire`\n\n:\n\n```\nrank   left   right----   ----   -----4      r      e52     Ġ      g79     l      d211    Ġg     o301    i      re1221   s      h3613   Ġgo    ld10676  sh     ire\n```\n\nAfter merging, the token strings are looked up in `encoder.json`\n\nwhich contains every token in GPT-2's vocabulary (256 base byte tokens and ~50k merged byte-sequence tokens).\n\n```\nbase byte symbols     token ID-----------------     --------a                     64b                     65Ġ  (space)            220\nmerged BPE pieces     token ID------------------    --------the                   1169ing                   278Ġhello                23748Ġworld                995\n```\n\nSo here's the complete path for `goldshire`\n\n:\n\n```\nĠ | g | o | l | d | s | h | i | r | e                                \\___/                                r + e, rank 4\nĠ | g | o | l | d | s | h | i | re  -> Ġg | o | l | d | s | h | i | re  -> Ġg | o | ld | s | h | i | re  -> Ġgo | ld | s | h | i | re  -> Ġgo | ld | s | h | ire  -> Ġgo | ld | sh | ire  -> Ġgold | sh | ire  -> Ġgold | shire\nĠgold -> 3869shire -> 10932\n```\n\nEach round chooses the valid pair with the lowest rank. A merge can create a new pair, so BPE cannot just scan from left to right.\n\n`goldshire`\n\ntakes eight merge rounds to become two IDs. The full sentence takes 24 rounds to become eight IDs.\n\n## Baseline Version\n\nGPT-2's [ encoder.py](https://github.com/openai/gpt-2/blob/master/src/encoder.py) ports fairly simply to Rust. I've used the same regex, byte map, merge ranks, and\n\n`encoder.json`\n\n. And it returns the complete `Vec<u32>`\n\n(same interface). I tried to keep the control flow roughly the same as the reference so it's easier to follow.\n\n``` php\npub fn encode(input: &str) -> Vec<u32> {    tokenizer().encode(input)}\nfn encode(&self, input: &str) -> Vec<u32> {    self.pattern.find_iter(input)        // Each match is one independent BPE region.        .flat_map(|item| self.merge(item.unwrap().as_str()))        // Final BPE pieces map to GPT-2 token IDs.        .map(|piece| self.token_ids[&self.symbols[piece]])        .collect()}\n```\n\nHere's the outer encode loop and the repeated merge loop:\n\n``` js\nloop {    let best = pieces.windows(2)        // The earliest vocab.bpe line has the best rank.        .filter_map(|pair| {            self.merge_ranks                .get(&(pair[0], pair[1]))                .map(|&(rank, result)| {                    (pair[0], pair[1], rank, result)                })        })        .min_by_key(|pair| pair.2);\n    let Some((first, second, _, result)) = best else {        return pieces;    };\n    // Apply this merge wherever the pair occurs.    pieces = merge_all(pieces, first, second, result);}\n```\n\nI verified the final ID sequence against the Python reference for a few different inputs to make sure it was correct. My benchmarks measure tokenizer time on an M1 Pro.\n\nMy Rust version keeps `encoder.py`\n\n's cache shape (byte-to-Unicode encoded regex match to merged BPE symbols) but unlike the reference's unbounded cache, I capped it at 256 LRU entries to approximate a bounded production cache. This cache is not persisted between `encode`\n\ncalls.\n\nI'll use the same two inputs throughout: the first 32 KiB of Moby-Dick and 8 MiB of React source code. I also use random ASCII and base64-like text in the background for adversarial checks for the merge-loop section.\n\nI take the median of nine calls after one untimed setup call.\n\n```\n                         Moby-Dick, 32 KiB       React, 8 MiBPython encoder.py             42.43 ms              0.7 MiB/sRust baseline                  7.01 ms              6.0 MiB/s\n```\n\nThis Rust port is a great place to start but it's not yet a clever tokenizer. The lowest hanging fruit to improve is the repeated whole-sequence scan in its merge loop.\n\n## Version 2: An Attempt to Merge Faster\n\nThe baseline version repeatedly scans every adjacent pair to find the lowest-ranked pair and then merges it and scans again. This is a direct port of the reference version, so while it's clear and easier to understand, it revisits most of a long match after every merge. `goldshire`\n\nneeds eight rounds despite producing only two final tokens.\n\nBelow are the first four merges from the real `goldshire`\n\ntrace. GPT-2 chooses the lowest-ranked valid adjacent pair (not the leftmost pair or the longest token). So `r+e`\n\n, which has rank 4, wins before the leading-space pair, even though it is near the end of the match.\n\nThe important bit to take away is that each merge can expose another candidate.\n\n```\nĠ | g | o | l | d | s | h | i | r | e                                \\___/                                r + e, rank 4\nĠ | g | o | l | d | s | h | i | re\\___/ Ġ + g, rank 52\nĠg | o | l | d | s | h | i | re         \\___/         l + d, rank 79\nĠg | o | ld | s | h | i | re\\____/Ġg + o, rank 211\n```\n\nInstead of doing a full rescan, we can use a priority queue of candidate pairs (to avoid looking up the matches which stayed the same) and a linked-list sequence of symbols (because a merge only needs to update its two new neighbours).\n\nI'll show the priority queue draining towards the end of `goldshire`\n\n's merges here:\n\n```\nround 5: [Ġgo]--[ld]--[s]--[h]--[ire]queue:   (s,h) 1221  (Ġgo,ld) 3613  (h,ire) 10439\nround 6: [Ġgo]--[ld]--[sh]--[ire]queue:   (Ġgo,ld) 3613  (sh,ire) 10676\nround 7: [Ġgold]--[sh]--[ire]queue:   (sh,ire) 10676\nround 8: [Ġgold]--[shire]queue:   empty (merges complete)\n```\n\nA min-priority queue selects the next pair type by its GPT-2 rank and an indexed linked list keeps neighbours available after a merge. Pair positions are checked again when popped because earlier merges can make queued entries stale.\n\nBut wait! The benchmarks say this *is not* a speed-up:\n\n```\n                         Moby-Dick, 32 KiB       React, 8 MiBRust baseline                  7.01 ms              6.0 MiB/sHeap and neighbours            9.25 ms              5.4 MiB/s\n```\n\nThe reference loop has bad-looking asymptotics per regex match but these matches are often small. I checked their sizes after trying the heap optimization. At this smaller scale, linear scans of contiguous memory are cheap while the heap adds pair maps, stale entries, node indirection, sorting, and branches. And all of this doesn't amortize well.\n\n```\nInput                  Matches       Average     P90      MaxMoby-Dick, 32 KiB      7,632         4.29 B      8 B      17 BReact, 8 MiB           1,862,860     4.50 B      9 B      89 B\n```\n\nHeaps aren't bad though. In fact, OpenAI's [tiktoken](https://github.com/openai/tiktoken) uses a separate heap and compact-state path for pieces that are at least 100 bytes.\n\n## Version 3: Use Bytes Directly\n\nMy V3 is inspired by [tiktoken](https://github.com/openai/tiktoken)'s public `src/lib.rs`\n\n. It is a small GPT-2-compatible reimplementation.\n\nWe can decode GPT-2's `encoder.json`\n\nentries to raw bytes once during setup. In GPT-2, IDs 0 through 255 are the base byte tokens. Learned merge results receive subsequent IDs in merge priority order. Among learned merge tokens, a lower ID means an earlier vocab.bpe merge so token ID minus 256 is that merge's zero-based rank.\n\nHopefully this comparison makes that a bit clearer:\n\n``` php\nbaseline:\nb\" goldshire\"  -> translate each byte into GPT-2's Unicode alphabet  -> Ġ | g | o | l | d | s | h | i | r | e  -> use vocab.bpe to choose merges between symbol pairs  -> Ġgold | shire  -> look up the finished symbols in encoder.json  -> [3869, 10932]\nV3:\nb\" goldshire\"  -> keep offsets into these original bytes  -> use one map: byte sequence -> token ID / merge rank  -> b\" gold\" | b\"shire\"  -> [3869, 10932]\n```\n\nWhile we're merging a match, any active parts are adjacent ranges in the unchanged input bytes, so their concatenation is already a borrowed contiguous byte slice. Looking that slice up in the decoded vocabulary tells us whether it is a legal GPT-2 merge candidate. Its ID gives the merge priority because of GPT-2's vocabulary construction.\n\n```\nvocab.bpe                     decoded encoder.json\nrank 4:     r + e             b\"re\"     -> token ID 260rank 52:    Ġ + g             b\" g\"     -> token ID 308rank 79:    l + d             b\"ld\"     -> token ID 335\nsmaller merged-token ID  <=>  earlier merge  <=>  higher priority\n```\n\nAnother benefit of this decoding is that sometimes we can go from a complete regex match (like `the`\n\nor `wow`\n\n) directly to the final token ID (in this case `1169`\n\nor `42773`\n\n).\n\n``` js\nlet bytes = piece.as_bytes();\n// A learned token needs no BPE work at all.if let Some(&token) = ranks.get(bytes) {    return vec![token];}\n```\n\nV3 also brings a new memory layout to minimize allocations.\n\nOn a BPE cache miss, V3 allocates a byte-to-Unicode cache key, a `Vec<Part>`\n\nfor surviving byte boundaries, cache storage for the merged result, and the output token IDs. The original input bytes are never moved or copied.\n\n```\nfixed input bytes\noffset:   0   1   2   3   4          | a | b | c | e |\ninitial boundary vector\npart:    [0]       [1]       [2]       [3]       [4]rank:  rank(\"ab\") rank(\"bc\") rank(\"ce\")  none      none                    ^                    lowest rank wins\ncurrent pieces:  a | b | c | e\n```\n\nMerging `b + c`\n\ndoesn't create a `b\"bc\"`\n\nstring. It removes the boundary at byte offset 2. The current pieces are always the ranges between surviving offsets.\n\n```\nmerge b + c\nbefore:  offsets [0] [1] [2] [3] [4]                          ^                          delete this boundary\nafter:   offsets [0] [1]     [3] [4]pieces:             a |  bc   | e\nnew candidates:bytes[0..3] = b\"abc\"       a + bcbytes[1..4] = b\"bce\"       bc + e\n```\n\nThe merge loop is then:\n\n- Find the smallest non-\n`u32::MAX`\n\nrank. - Update the candidate at that boundary and the one on its left.\n- Remove the boundary to its right. The vector shifts later entries in place.\n- Repeat.\n- Stop when no part has a valid rank, which means no two remaining neighbouring byte ranges form a vocabulary token.\n\n```\nstruct Part {    start: usize,    rank: u32, // Candidate merged-token ID, or u32::MAX.}\n// Allocate once. The input bytes never change.let mut parts = initial_parts(bytes);\nwhile let Some((index, _)) = lowest_rank(&parts) {    // Recheck the two candidates that will touch after removal.    update_rank(bytes, &mut parts, index.saturating_sub(1));    update_rank(bytes, &mut parts, index);\n    // Delete one boundary. Vec shifts entries but does not reallocate.    parts.remove(index + 1);}\n```\n\nAfter all eight merges in `goldshire`\n\n, the boundary vector is `[0, 5, 10]`\n\n, which defines the final byte ranges. The output `Vec<u32>`\n\nis filled by borrowing each final byte slice as a key in the decoded vocabulary map.\n\n```\noriginal bytes:  b\" goldshire\"final offsets:   [0]       [5]       [10]final ranges:     \\________/ \\________/                  b\" gold\"   b\"shire\"                     |           |                     v           v                   3869        10932\noutput: Vec<u32> = [3869, 10932]\n```\n\nI also copied tiktoken's use of `FxHashMap`\n\n(over Rust's general-purpose hash map).\n\n```\nVariant                         Moby-Dick, 32 KiB    React, 8 MiB-----------------------------   -----------------   -----------Python encoder.py                42.43 ms            0.7 MiB/sRust baseline                     7.01 ms            6.0 MiB/sHeap and neighbours               9.25 ms            5.4 MiB/sV3 check vocabulary first         3.09 ms           9.4 MiB/sV3 check cache first              3.23 ms            8.8 MiB/stiktoken-rs r50k_base             2.15 ms           11.7 MiB/s\n```\n\nV3 vocabulary-first is roughly 2.3x faster than the Rust baseline for Moby-Dick and 1.6x faster on React. Direct vocabulary lookup and compact contiguous state beat a complicated global data structure for this workload.\n\nAfter reducing merge work, the regex part becomes the largest single measured stage, and cache and output conversion also become more visible.\n\n```\nstage              V1 Moby  V2 Moby  V3 Moby  V1 React  V2 React  V3 Reactregex                 29%      20%      55%       39%       35%       55%byte mapping           4%       3%       9%        6%        5%        9%cache                  3%       2%      10%        4%        4%        8%BPE merging           51%      69%       9%       40%       47%        8%lookup/output         13%       6%      17%       11%       10%       20%\n```\n\nSo where does LLM tokenization spend its time? The answer for GPT-2 (some of this should generalize) is regex matching, merging, and look-ups.\n\nTokenization is not slow because any one step is complex but rather because it needs to perform a huge number of tiny regex, look-up, and merge operations on very small pieces of text.\n\n## Ending Thoughts\n\nSome product features need a rough size instead of token IDs. Fixed estimates like `input.len() / N`\n\nare nearly free because they don't need to tokenize at all.\n\n```\nestimated tokens = input bytes / N\n```\n\nYou can even choose `N`\n\nbased on what you know about the input, e.g. whether it is a code file.\n\nAll the versions of my tokenizer received a complete string but a service will often receive a request body in chunks, or an application may add to a prompt incrementally. But chunk counts are not additive! A later chunk can change a regex region or finish a BPE merge from the previous chunk:\n\n```\ncount(\" goldshi\") + count(\"re\")  = 2 + 1  = 3\ncount(\" goldshire\")  = 2\ntokens(\"hello \") + tokens(\"world\")  = [31373, 220] + [6894]\ntokens(\"hello world\")  = [31373, 995]\n```\n\nI didn't dig into streaming tokenization for this article but it seems like an interesting problem. A safe tail is not only a BPE problem because future bytes can also alter the pre-tokenizer's regex boundaries. [tiktoken](https://github.com/openai/tiktoken) has `encode_with_unstable`\n\nwhich encodes a string into stable tokens and *possible completion* sequences.\n\nIn terms of performance takeaways, it's a common one: contiguous scans are cheap! One experiment I ran but didn't write-up was using [Rayon](https://github.com/rayon-rs/rayon) to handle regex matches in parallel. It was a little faster for some inputs but not dramatically (and the performance was a bit unpredictable). While complete regex matches can be processed independently, the matches are so small that coordination/scheduling tends to dominate.\n\nMy measurements and findings are for GPT-2's byte-level BPE, regex, and vocabulary construction. Modern tokenizers can use different vocabularies, normalizers, regexes, and special-token rules. But the underlying work is roughly the same.\n\nThe source code and benchmarks can be found here: [github.com/healeycodes/gpt-2-tokenizer](https://github.com/healeycodes/gpt-2-tokenizer). My email is on my [home page](https://healeycodes.com). Let me know if I got something wrong, or if I missed an important optimization :)", "url": "https://wpnews.pro/news/what-makes-llm-tokenization-slow", "canonical_source": "https://healeycodes.com/what-makes-llm-tokenization-slow", "published_at": "2026-09-02 09:24:42+00:00", "updated_at": "2026-09-02 09:52:22.128208+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing"], "entities": ["GPT-2", "OpenAI", "Rust"], "alternates": {"html": "https://wpnews.pro/news/what-makes-llm-tokenization-slow", "markdown": "https://wpnews.pro/news/what-makes-llm-tokenization-slow.md", "text": "https://wpnews.pro/news/what-makes-llm-tokenization-slow.txt", "jsonld": "https://wpnews.pro/news/what-makes-llm-tokenization-slow.jsonld"}}