TokenBpe A editorial on how tokenizers can be sped up. A developer has released TokenBpe, a BPE tokenizer written from scratch in C++20 that uses SWAR byte classification, dual-cursor splitting, per-thread caching and a linked-list merge to speed up LLM tokenization. In an audited three-way benchmark on a 1.06 MB Shakespeare corpus with a GPT-2 vocabulary on an Intel i5-11260H, TokenBpe encoded at 239.98 MB/s across 12 threads versus 45.83 MB/s for tiktoken and 12.08 MB/s for Hugging Face tokenizers, roughly 5.2× and 19.9× faster respectively, and decoded at 341.01 MB/s single-threaded. The author notes the SWAR pre-tokenizer approximates GPT-2's regex, producing token counts about 1.9% off tiktoken's, though roundtrips remain exact. Every LLM pipeline starts with a tokenizer and almost no one thinks about it. The model gets the headlines, the tokenizer is quietly burning microseconds per token per request, forever. If you're serving millions of requests or just training on a corpus that doesn't fit in memory that "quietly" adds up. This post is a walk through TokenBpe, a BPE tokenizer that I've written from the ground up in C++20. It covers what the various components are, how each one works, what specific techniques give you a performance gain and why. All of this is based on the actual code include/tokenizer/, src/, utils/ and the benchmark figures are from a properly audited 3 way comparison against OpenAI's tiktoken Rust and Hugging Face tokenizers. Headline numbers Shakespeare corpus, 1.06 MB, GPT-2 vocabulary, 12 hardware threads, Intel i5-11260H : | Operation | TokenBpe | tiktoken | HF tokenizers | |---|---|---|---| | Encode, 12T zero-copy | 239.98 MB/s | 45.83 MB/s | 12.08 MB/s | | Encode, 1T | 34.22 MB/s | 11.55 MB/s | 2.11 MB/s | | Decode, 1T zero-copy | 341.01 MB/s | 117.34 MB/s | 24.93 MB/s | ~5.2× tiktoken and ~19.9× HF on 12-thread encoding. And ~6.1× scaling from 1 to 12 threads natively. Let's open the hood. BPE encoding has two phases: Decoding is trivial really: just map each token id to its bytes and concatenate. The naive implementation of this is fine. The real question is where the time goes and the answer, as ever, is everywhere, a little bit at a time. So TokenBpe attacks every layer: text ─▶ SWAR pre-tokenizer ─▶ dual-cursor split ─▶ per-thread cache ─▶ linked-list merge ─▶ token ids with a persistent thread pool under the whole thing, a flat binary model format and a zero copy NumPy boundary on top. The technique. SWAR SIMD Within A Register does the 8 bytes per 64 bit word classification using pure integer arithmetic. No SSE/AVX intrinsics at all, it's all done in the 64 bit ALU. // swar.hpp simplified // All 8 lanes are bytes; HIGH BITS is OR'd in before subtracting // min so no lane can borrow from its neighbour. bool all bytes in range uint64 t v, uint8 t min, uint8 t max ; bool all ascii lower uint64 t v ; // a-z bool all ascii upper uint64 t v ; // A-Z bool all ascii letters uint64 t v ; // both, via v | 0x20 bool all ascii digits uint64 t v ; // 0-9 bool all bytes equal uint64 t v, uint8 t b ; The range check is written very carefully indeed: it rejects the high bit bytes first, then ORs in HIGH BITS prior to subtracting min, so a lane can never borrow from its neighbour. load64 does an unaligned 64-bit load via memcpy which ends up as a single instruction. The splitter pre tokenizer.hpp then scans: all bytes equal ; 's , 't , 'm , 'd , 're , 've , 'll ; And crucially next token returns a std::string view into the original buffer. No pre-token is ever materialised as a string. The caller owns the text lifetime, the splitter just points into it. The speed payoff. Byte classification goes from one branch per byte to one instruction per eight bytes for the majority of the scan. No regex engine is actually invoked at all. The only caveat here is that it's an approximation of GPT-2's regex exotic Unicode whitespace can have different effects on splitting , which is why the token counts on the Shakespeare corpus differ by ~1.9% from tiktoken. Roundtrips are always exact though. The problem . Pre-tokenising left to right is a single dependency chain: scan, emit, scan, emit. The CPU can't get ahead of itself as each step is dependent on where the last one left off. The technique. dual cursor.hpp finds a safe split point a space or newline within ±128 bytes of the midpoint, searching backwards first inputs under 256 bytes aren't split and runs two LexerCursor s alternately in one fused loop : // dual cursor.hpp — sketch find safe split boundary text ; // near midpoint, whitespace-aligned LexerCursor a{begin, split}, b{split, end}; while a.has more || b.has more { if a.has more emit next token a , out a ; if b.has more emit next token b , out b ; } Each cursor's dependency chain is independent so the superscalar core has two instruction streams to work on. Output order is preserved by writing the first half into out a and the second half into out b and then concatenating. What it is NOT: threading. It's a fused, allocation-free iteration that also eschews the need to materialise a std::vector std::string view of pre-tokens. The technique. encode chunk uncached to buffer the heart of src/encoder.cpp represents the pre-token as a thread-local array of nodes: struct ListNode { TokenId token; int32 t prev; int32 t next; }; The algorithm: merge ranks , keep the one with the globally lowest rank not the first one encountered, training order takes priority regardless of position . If none, stop. The speed payoff. No string allocations at all during merging, ever. The merged bytes were already computed by add merge when the model was built. The cost is an O n scan per merge and for pre-tokens which are almost always very short that's a very reasonable trade. This is also where the TokenPairHash comes in: a splitmix64 style mixer on first << 32 | second, giving us unordered map lookups that won't pathologically collide. The technique. Each thread gets an 8 192-slot, direct-mapped cache: static assert sizeof CacheSlot == 64 ; // exactly one cache line // slot = fnv1a64 word & 0x1FFF // slot = { hash, word len, token count, char word 22 , TokenId tokens 8 } The speed payoff. A cache hit bypasses the scratchpad, the list construction, the merge scans and the hash-map probes of the pair table it's just a single line lookup and a bulk insert. The 512kib/thread footprint remains in L2 on most cores. Decoding looks up decode table id .size to get the total output size, calls resize once and then memcpy s each token's bytes forward. The table is id-indexed so decode never touches a hash map. Any ids outside the table are silently skipped a corrupt stream just degrades rather than aborting . Result: 341mb/s, ~2.9x tiktoken, ~13.7x HF. Spawning a thread per batch kills you twice: on thread creation and on cold thread local state. TokenBpe uses a PersistentThreadPool: hardware concurrency - 1 workers plus the calling thread , work distributed via a shared atomic cursor fetch add a dynamic schedule with no per-item queue. The workers keep their TokenCache and ThreadScratchpad warm across calls for the life of the process. Batches go through parallel tokenize flat which returns a single contiguous token array plus a prefix sum offsets array a FlatBatch such that chunk i is in tokens offsets i .. offsets i+1 . No per-chunk vectors, no copies. Honest limitation: one shared task slot, non-reentrant, threads parameter is advisory. It's a very simple pool really, does one thing and does it well. tolist The nanobind FFI in utils/bindings.cpp is where the vast majority of the "fast" tokenizers just quietly fall apart, they're fast at computing, then they copy everything into Python objects. TokenBpe: encode flat / encode batch flat as numpy.uint32 ndarrays whose std::vector is moved to the heap and owned by an nb::capsule — ownership transfers to NumPy, no Measured effect: ndarray encode at 239.98 MB/s vs 137.40 MB/s for the equivalent list list int path 1.75x purely from the boundary . save / load use a compact, order-independent binary: Header : magic "BPE1" | version u8 | vocab size u32 | merge count u32 Vocabulary : vocab size × { id u32 | len u32 | bytes len } Merges : merge count × { left u32 | right u32 | rank u32 | result u32 } Load validates magic/version, clears all tables and the thread-local cache , then rebuilds byte to token id from length-1 vocab entries essential for imported GPT-2, whose byte tokens are permuted , merge ranks , and the flat decode table. A 50 257-token / 49 992-merge GPT-2 model is ~1.5 MB and loads in microseconds. There's also a trainer frequency-weighted, recompute-based, deliberately simple and a Hugging Face importer convert hf to bpe.py that reverses GPT-2's byte-to-unicode map. Every technique maps to a specific cost it eliminates: | Technique | Eliminates | |---|---| | SWAR pre-tokenization | Regex engine overhead; 8× fewer classification steps | | string view pre-tokens | String materialization + allocator traffic | | Dual cursor | Single dependency chain; enables ILP | | Linked-list merge | O n² vector erase-and-shift; string building | | Global lowest-rank selection | Correctness of merge priority a subtle bug source, not just speed | | Direct-mapped 64 B cache | Scratchpad setup + merge work for common pre-tokens | | Thread-local scratchpad/cache | Cross-thread synchronization + allocation | | Persistent pool + atomic cursor | Thread spawn cost; queue allocation | | Flat decode table + single resize | Hash lookups; multiple allocations | | Flat batch + prefix-sum offsets | Per-chunk vector overhead | | Zero-copy ndarray + GIL release | Python object overhead; interpreter serialization | | BPE1 binary model | Parse overhead at load time | | -O3 -march=native -flto | The compiler's chances to help | The big lesson: no single trick wins . The speedup is the product of a dozen multiplicative factors, boundary copies, allocations, cache misses, dependency chains, each small, all compounding. RAYON NUM THREADS to the C++ pool's thread count. fast bpe and tiktoken fast bpe C++20 Encode 12T 239.98 MB/s 74.80 Mtok/s zero-copy ndarray fast bpe C++20 Encode 12T 137.40 MB/s 42.83 Mtok/s Python list list int tiktoken Rust Encode 12T 45.83 MB/s 14.56 Mtok/s HF tokenizers Encode 12T 12.08 MB/s 3.84 Mtok/s fast bpe C++20 Decode 1T 341.01 MB/s 106.30 Mtok/s zero-copy ndarray Native C++ suite: 41.81 MB/s single-threaded → 254.88 MB/s on 12 threads = 6.10× scaling. Not a perfect 12× memory bandwidth and the single split point in the dual cursor, among other things. Caveats to state in the post: the pre-tokenizer is an approximation ~1.9% fewer tokens on this corpus , roundtrips are exact, and fast bpe vs tiktoken is not a byte-identical id comparison. git clone