{"slug": "tokenbpe-a-editorial-on-how-tokenizers-can-be-sped-up", "title": "TokenBpe A editorial on how tokenizers can be sped up.", "summary": "A developer has released TokenBpe, a BPE tokenizer written from scratch in C++20 that uses SWAR byte classification, dual-cursor splitting, per-thread caching and a linked-list merge to speed up LLM tokenization. In an audited three-way benchmark on a 1.06 MB Shakespeare corpus with a GPT-2 vocabulary on an Intel i5-11260H, TokenBpe encoded at 239.98 MB/s across 12 threads versus 45.83 MB/s for tiktoken and 12.08 MB/s for Hugging Face tokenizers, roughly 5.2× and 19.9× faster respectively, and decoded at 341.01 MB/s single-threaded. The author notes the SWAR pre-tokenizer approximates GPT-2's regex, producing token counts about 1.9% off tiktoken's, though roundtrips remain exact.", "body_md": "Every LLM pipeline starts with a tokenizer and almost no one thinks about it. The model gets the headlines, the tokenizer is quietly burning microseconds per token per request, forever. If you're serving millions of requests or just training on a corpus that doesn't fit in memory that \"quietly\" adds up.\n\nThis post is a walk through TokenBpe, a BPE tokenizer that I've written from the ground up in C++20. It covers what the various components are, how each one works, what specific techniques give you a performance gain and why. All of this is based on the actual code (include/tokenizer/, src/, utils/) and the benchmark figures are from a properly audited 3 way comparison against OpenAI's tiktoken (Rust) and Hugging Face tokenizers.\n\n**Headline numbers** (Shakespeare corpus, 1.06 MB, GPT-2 vocabulary, 12 hardware threads, Intel i5-11260H):\n\n| Operation | TokenBpe | tiktoken | HF tokenizers | \n|---|---|---|---|\n| Encode, 12T (zero-copy) | **239.98 MB/s** | 45.83 MB/s | 12.08 MB/s | \n| Encode, 1T | **34.22 MB/s** | 11.55 MB/s | 2.11 MB/s | \n| Decode, 1T (zero-copy) | **341.01 MB/s** | 117.34 MB/s | 24.93 MB/s | \n\n~5.2× tiktoken and ~19.9× HF on 12-thread encoding. And ~6.1× scaling from 1 to 12 threads natively. Let's open the hood.\n\nBPE encoding has two phases:\n\nDecoding is trivial really: just map each token id to its bytes and concatenate.\n\nThe naive implementation of this is fine. The real question is where the time goes and the answer, as ever, is everywhere, a little bit at a time. So TokenBpe attacks every layer:\n\n```\ntext ─▶ SWAR pre-tokenizer ─▶ dual-cursor split ─▶ per-thread cache ─▶ linked-list merge ─▶ token ids\n```\n\nwith a persistent thread pool under the whole thing, a flat binary model format and a zero copy NumPy boundary on top.\n\n**The technique.** SWAR (SIMD Within A Register) does the 8 bytes per 64 bit word classification using pure integer arithmetic. No SSE/AVX intrinsics at all, it's all done in the 64 bit ALU.\n\n```\n// swar.hpp (simplified)\n// All 8 lanes are bytes; HIGH_BITS is OR'd in before subtracting\n// min so no lane can borrow from its neighbour.\nbool all_bytes_in_range(uint64_t v, uint8_t min, uint8_t max);\nbool all_ascii_lower(uint64_t v);   // [a-z]\nbool all_ascii_upper(uint64_t v);   // [A-Z]\nbool all_ascii_letters(uint64_t v); // both, via v | 0x20\nbool all_ascii_digits(uint64_t v);  // [0-9]\nbool all_bytes_equal(uint64_t v, uint8_t b);\n```\n\nThe range check is written very carefully indeed: it rejects the high bit bytes first, then ORs in HIGH_BITS prior to subtracting min, so a lane can never borrow from its neighbour. load64 does an unaligned 64-bit load via memcpy which ends up as a single instruction.\n\nThe splitter (`pre_tokenizer.hpp`) then scans:\n\n`all_bytes_equal`;`'s`, `'t`, `'m`, `'d`, `'re`, `'ve`, `'ll`;\nAnd crucially next_token() returns a std::string_view into the original buffer. No pre-token is ever materialised as a string. The caller owns the text lifetime, the splitter just points into it.\n\n**The speed payoff.** Byte classification goes from one branch per byte to one instruction per eight bytes for the majority of the scan. No regex engine is actually invoked at all. The only caveat here is that it's an approximation of GPT-2's regex (exotic Unicode whitespace can have different effects on splitting), which is why the token counts on the Shakespeare corpus differ by ~1.9% from tiktoken. Roundtrips are always exact though.\n\n**The problem**. Pre-tokenising left to right is a single dependency chain: scan, emit, scan, emit. The CPU can't get ahead of itself as each step is dependent on where the last one left off.\n\n**The technique.** `dual_cursor.hpp` finds a safe split point a space or newline within ±128 bytes of the midpoint, searching backwards first (inputs under 256 bytes aren't split) and runs **two `LexerCursor` s alternately in one fused loop**:\n\n```\n// dual_cursor.hpp — sketch\nfind_safe_split_boundary(text);          // near midpoint, whitespace-aligned\nLexerCursor a{begin, split}, b{split, end};\nwhile (a.has_more() || b.has_more()) {\n    if (a.has_more()) emit(next_token(a), out_a);\n    if (b.has_more()) emit(next_token(b), out_b);\n}\n```\n\nEach cursor's dependency chain is independent so the superscalar core has two instruction streams to work on. Output order is preserved by writing the first half into out_a and the second half into out_b and then concatenating.\n\n**What it is NOT:** threading. It's a fused, allocation-free iteration that also eschews the need to materialise a std::vector[std::string_view] of pre-tokens.\n\n**The technique.** `encode_chunk_uncached_to_buffer()` (the heart of `src/encoder.cpp`) represents the pre-token as a thread-local array of nodes:\n\n```\nstruct ListNode { TokenId token; int32_t prev; int32_t next; };\n```\n\nThe algorithm:\n\n`merge_ranks_`, keep the one with the globally lowest rank (not the first one encountered, training order takes priority regardless of position). If none, stop.\n**The speed payoff.** No string allocations at all during merging, ever. The merged bytes were already computed by add_merge() when the model was built. The cost is an O(n) scan per merge and for pre-tokens (which are almost always very short) that's a very reasonable trade.\n\nThis is also where the TokenPairHash comes in: a splitmix64 style mixer on first << 32 | second, giving us unordered_map lookups that won't pathologically collide.\n\n**The technique.** Each thread gets an 8 192-slot, direct-mapped cache:\n\n```\nstatic_assert(sizeof(CacheSlot) == 64);  // exactly one cache line\n// slot = fnv1a64(word) & 0x1FFF\n// slot = { hash, word_len, token_count, char word[22], TokenId tokens[8] }\n```\n\n**The speed payoff.** A cache hit bypasses the scratchpad, the list construction, the merge scans and the hash-map probes of the pair table it's just a single line lookup and a bulk insert. The 512kib/thread footprint remains in L2 on most cores.\n\nDecoding looks up `decode_table_[id].size()` to get the total output size, calls `resize()` once and then `memcpy` s each token's bytes forward. The table is id-indexed so decode never touches a hash map. Any ids outside the table are silently skipped (a corrupt stream just degrades rather than aborting). Result: 341mb/s, ~2.9x tiktoken, ~13.7x HF.\n\nSpawning a thread per batch kills you twice: on thread creation and on cold thread local state. TokenBpe uses a PersistentThreadPool: `hardware_concurrency() - 1` workers **plus the calling thread**, work distributed via a shared atomic cursor (`fetch_add`) a dynamic schedule with no per-item queue. The workers keep their TokenCache and ThreadScratchpad warm across calls for the life of the process.\n\nBatches go through parallel_tokenize_flat() which returns a single contiguous token array plus a prefix sum offsets array (a FlatBatch) such that chunk i is in tokens[offsets[i] .. offsets[i+1]]. No per-chunk vectors, no copies.\n\nHonest limitation: one shared task slot, non-reentrant, `threads` parameter is advisory. It's a very simple pool really, does one thing and does it well.\n\n`tolist()`\nThe nanobind FFI (in utils/bindings.cpp) is where the vast majority of the \"fast\" tokenizers just quietly fall apart, they're fast at computing, then they copy everything into Python objects. TokenBpe:\n\n`encode_flat` / `encode_batch_flat` as `numpy.uint32` ndarrays whose `std::vector` is moved to the heap and owned by an `nb::capsule` — ownership transfers to NumPy, no Measured effect: ndarray encode at 239.98 MB/s vs 137.40 MB/s for the equivalent `list[list[int]]` path **1.75x purely from the boundary**.\n\n`save()`/` load()` use a compact, order-independent binary:\n\n```\nHeader     : magic \"BPE1\" | version u8 | vocab_size u32 | merge_count u32\nVocabulary : vocab_size × { id u32 | len u32 | bytes[len] }\nMerges     : merge_count × { left u32 | right u32 | rank u32 | result u32 }\n```\n\nLoad validates magic/version, clears all tables **and the thread-local cache**, then rebuilds `byte_to_token_id` (from length-1 vocab entries essential for imported GPT-2, whose byte tokens are permuted), `merge_ranks_`, and the flat decode table. A 50 257-token / 49 992-merge GPT-2 model is ~1.5 MB and loads in microseconds. There's also a trainer (frequency-weighted, recompute-based, deliberately simple) and a Hugging Face importer (`convert_hf_to_bpe.py`) that reverses GPT-2's byte-to-unicode map.\n\nEvery technique maps to a specific cost it eliminates:\n\n| Technique | Eliminates | \n|---|---|\n| SWAR pre-tokenization | Regex engine overhead; 8× fewer classification steps | \n| `string_view` pre-tokens | String materialization + allocator traffic | \n| Dual cursor | Single dependency chain; enables ILP | \n| Linked-list merge | O(n²) vector erase-and-shift; string building | \n| Global lowest-rank selection | Correctness of merge priority (a subtle bug source, not just speed) | \n| Direct-mapped 64 B cache | Scratchpad setup + merge work for common pre-tokens | \n| Thread-local scratchpad/cache | Cross-thread synchronization + allocation | \n| Persistent pool + atomic cursor | Thread spawn cost; queue allocation | \n| Flat decode table + single `resize` | Hash lookups; multiple allocations | \n| Flat batch + prefix-sum offsets | Per-chunk vector overhead | \n| Zero-copy ndarray + GIL release | Python object overhead; interpreter serialization | \n| `BPE1` binary model | Parse overhead at load time | \n| `-O3 -march=native -flto` | The compiler's chances to help | \n\nThe big lesson: **no single trick wins**. The speedup is the product of a dozen multiplicative factors, boundary copies, allocations, cache misses, dependency chains, each small, all compounding.\n\n`RAYON_NUM_THREADS` to the C++ pool's thread count.`fast_bpe` and `tiktoken` \n\n```\nfast_bpe (C++20)   Encode 12T   239.98 MB/s   74.80 Mtok/s   zero-copy ndarray\nfast_bpe (C++20)   Encode 12T   137.40 MB/s   42.83 Mtok/s   Python list[list[int]]\ntiktoken (Rust)    Encode 12T    45.83 MB/s   14.56 Mtok/s\nHF tokenizers      Encode 12T    12.08 MB/s    3.84 Mtok/s\nfast_bpe (C++20)   Decode 1T    341.01 MB/s  106.30 Mtok/s   zero-copy ndarray\n```\n\nNative C++ suite: 41.81 MB/s single-threaded → 254.88 MB/s on 12 threads = **6.10×** scaling. (Not a perfect 12× memory bandwidth and the single split point in the dual cursor, among other things.)\n\nCaveats to state in the post: the pre-tokenizer is an approximation (~1.9% fewer tokens on this corpus), roundtrips are exact, and `fast_bpe` vs tiktoken is not a byte-identical id comparison.\n\n```\ngit clone <https://github.com/Unica2804/Token_BPE.git> && cd BPE_Tokenizer\npython -m venv .venv && source .venv/bin/activate\npip install nanobind numpy\n# optional, for the 3-way benchmark:\npip install tiktoken tokenizers\n\ncmake -S . -B build\ncmake --build build -j\nctest --test-dir build --output-on-failure   # 10 tests, 52 assertions\n```\n\nArtifacts in `build/`: `libbpe_core.a`, `tokenizer_cli`, `tokenizer_bench`, `tokenizer_tests`, `fast_bpe.*.so`.\n\n```\npython utils/convert_hf_to_bpe.py \\\n    --vocab  gpt2_tokenizer/vocab.json \\\n    --merges gpt2_tokenizer/merges.txt \\\n    --out    gpt2.bpe\npython\nimport sys\nsys.path.append(\"./build\")\nimport fast_bpe\n\nenc = fast_bpe.BpeEncoder()\nenc.load(\"gpt2.bpe\")\n\nenc.encode(\"Hello, world!\")          # -> list[int]\nenc.encode_flat(\"Hello, world!\")     # -> numpy.uint32 ndarray (zero-copy)\n\ntokens, offsets = enc.encode_batch_flat([\"line 1\", \"line 2\"])\n# chunk i -> tokens[offsets[i]:offsets[i+1]]\n\nenc.decode(tokens)                   # -> str\n# MUST run from repo root\n.venv/bin/python benchmark.py --chunk-kb 64 --warmup 3 --runs 9\n./build/tokenizer_bench\n```\n\n*Built with C++20, nanobind, Catch2. Code, benchmarks, and the audited comparison script are in the repo.*", "url": "https://wpnews.pro/news/tokenbpe-a-editorial-on-how-tokenizers-can-be-sped-up", "canonical_source": "https://dev.to/unica_2804/tokenbpe-a-editorial-on-how-tokenizers-can-be-sped-up-57j5", "published_at": "2026-11-15 14:44:55+00:00", "updated_at": "2026-11-15 14:49:17.018548+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["TokenBpe", "tiktoken", "OpenAI", "Hugging Face", "Intel", "GPT-2"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/tokenbpe-a-editorial-on-how-tokenizers-can-be-sped-up", "markdown": "https://wpnews.pro/news/tokenbpe-a-editorial-on-how-tokenizers-can-be-sped-up.md", "text": "https://wpnews.pro/news/tokenbpe-a-editorial-on-how-tokenizers-can-be-sped-up.txt", "jsonld": "https://wpnews.pro/news/tokenbpe-a-editorial-on-how-tokenizers-can-be-sped-up.jsonld"}}