The AI industry pours engineering effort into GPU kernels and treats CPU preprocessing as an afterthought. Gigatoken shows how much performance that neglect can leave on the table.
In my first Gigatoken post, I tested its 1,000x claim against my tokenizer workload. I did not reproduce 1,000x, but the tokenizer core ran 30x to 40x faster, including 37.6x on a million-token request split into 1,024 segments. I wanted to understand where that gain came from.
I followed the path Gigatoken optimises most aggressively: tokenizers that apply BPE (byte pair encoding) to UTF-8 bytes, including GPT-2 and the tiktoken family. This is the path behind its headline throughput and the one exercised by my benchmark. The project has no technical paper yet, so what follows is my interpretation of Marcel Rød’s source comments, optimisation diary, profiling reports and commit history.Sources: the Gigatoken revision benchmarked and its profiling campaign. Gigatoken describes SentencePiece as less optimised and does not support WordPiece.Gigatoken's README says its optimised path is BPE; SentencePiece is less optimised and WordPiece unsupported.
How BPE turns bytes into tokens
The tokenizers in this article turn UTF-8 text into a sequence of integers. Their vocabularies map byte sequences to token IDs. A token might represent a whole word, part of a word, punctuation or a single byte. It is an entry in the vocabulary, not necessarily a word.
Encoding is a pipeline rather than one long merge loop. Pretokenisation is the coarse split immediately before BPE: it divides the continuous byte stream into spans using model-specific rules, commonly expressed as a regular expression. Each intermediate span is a pretoken. It is input to BPE, not a final model token. For a GPT-2-style tokenizer:
" token 42!" → [" token", " 42", "!"]
The hard boundaries also explain why the spans are independent. BPE is forbidden to merge across them, so a merge inside one pretoken cannot create or remove a candidate pair in another. The encoder can process each pretoken separately, then concatenate their token IDs in the original order.
Inside one pretoken, BPE starts from bytes or initial symbols. Its merge table was learned when the tokenizer was trained and is fixed during encoding. Each legal adjacent pair has a numeric rank, and the lower rank wins. If lo
is the best-ranked adjacent pair, the symbol sequence changes from [l] [o] [w] [e] [r]
to [lo] [w] [e] [r]
. The encoder looks up the affected neighbouring pairs again, chooses the next best merge and repeats until no legal merge remains. The surviving symbols map to token IDs.See the original BPE-for-subwords paper and OpenAI's compact educational implementation.
What Gigatoken changes inside the Rust core
Traditional Python tokenizer APIs inspect Python input objects and assemble Python-facing results; Gigatoken can read borrowed byte buffers and assemble a flat token buffer in Rust. The service I benchmarked already uses a Rust tokenizer core, so I exclude those API-boundary gains and examine the CPU and memory work inside tokenisation.Moving corpus splitting across the Python/Rust boundary was 11% to 16% faster with identical token IDs, but this isolates input handling, not output materialisation or the full native-versus-compatibility gap.
I group the Rust-core changes into four families: 64-byte boundary classification, cached pretoken encodings, compact pair-rank tables for misses and parallel work split at safe pretoken boundaries. They cover the main hot path for the BPE tokenizers in scope but are not exhaustive or equally responsible for the speed-up. The measurements use different machines, cache states and controls, so each result should be read against its own baseline.
1. Pretokenise 64 bytes at a time
A general-purpose regex engine finds one matching span after another. Gigatoken’s mask scanner instead asks which positions in a block of 64 input bytes begin pretokens. The block is only a unit of computation, not a pretoken boundary.
The first classification pass handles ASCII, where one byte represents one character and simple byte comparisons can identify letters, digits, spaces, newlines, apostrophes and other categories. SIMD performs these comparisons across several byte positions at once. Each position is called a lane, and each lane holds one byte here. Four 16-byte NEON loads cover the batch on ARM, two 32-byte AVX2 loads cover it on x86, and AVX-512 can load all 64 bytes at once. The comparisons become separate 64-bit class masks, with one bit for each input byte. Bytes at or above 0x80
are marked for Unicode handling rather than assigned a character class in this pass.Source: the mask-scanner architecture and SIMD front ends.
Gigatoken does not compile arbitrary regexes into this form. Its recognises a fixed set of tokenizer patterns and selects code written for that pattern. For each mask-scanner family, the Rust implementation expresses the regex’s boundary rules as operations over the class masks. The code treats each mask as a row of 64 on/off positions. A shift slides one row left or right so each byte lines up with its neighbour. AND keeps positions where two conditions are true, OR combines alternatives, and NOT selects positions outside a class. SIMD has finished once it creates the masks; ordinary 64-bit integer operations then combine them into boundary bits.
The author’s detailed optimisation notes and isolated scanner measurements use GPT-2’s r50k pretokenizer, so I use the same worked example here.The r50k module notes document the scalar path and isolated scanner setup. Under r50k’s rules, a leading space can join the letter, number or punctuation run after it. In ·go·42!
, where ·
is a space byte, shifting the letter and digit masks supplies the previous-position relationship for every byte at once. The resulting start bits identify ·go
, ·42
and !
.
Bit-parallel regex evaluation and SIMD block classification predate Gigatoken. Its contribution is specialising those techniques for the fixed patterns used by model tokenizers.The techniques predate Gigatoken: see bit-parallel regex matching and simdjson's SIMD classification. Gigatoken specialises them for tokenizer patterns.
The mask scanner must reproduce the regex’s boundaries. A pretoken can cross a 64-byte edge, so the scanner carries information from the preceding character and looks beyond the right edge where a rule needs it. Each batch returns the starts it can prove and marks any uncertain stretch as a bad zone for exact re-derivation. A bad zone is not necessarily malformed text; it means only that the block calculation cannot safely settle every boundary there.
Differential tests compare the combined mask and fallback output with the reference across crafted edge cases, 4,000 generated inputs and OpenWebText samples.The r50k module includes edge-case, fuzz and OpenWebText differential tests.
About 21% of batches in OpenWebText, a web-text corpus, contain at least one non-ASCII byte. UTF-8 breaks the ASCII pass’s one-byte-per-character assumption: one character can occupy several bytes, and those individual bytes do not reveal whether the character is a letter, number or whitespace. The SIMD pass therefore records their positions in a non-ASCII-byte mask for a second classification pass.
The second pass completes the same 64-position masks. For this r50k path, Gigatoken finds each UTF-8 lead byte, decodes its code point and uses a packed table of about 272 KiB to classify the character as a letter, number, whitespace or other. It stamps that class across every byte of the UTF-8 character, preventing a continuation byte from becoming a false boundary, then adds those results to the masks used by the tokenizer’s boundary rules.
Any region the masks cannot settle falls back to Gigatoken’s scalar walker. It uses ordinary integer instructions rather than SIMD lanes and advances from one span boundary to the next instead of classifying a whole 64-byte block. This fallback also handles CPUs without the required SIMD features, the incomplete tail of a buffer and bad zones such as invalid UTF-8 or ambiguous batch-edge cases. For r50k on the OpenWebText sample, only about 0.4% of batches require scalar re-derivation.See Gigatoken's packed Unicode tables and r50k scalar fallback.
When boundaries are requested one at a time, the scanner finds the lowest set bit with trailing_zeros
and clears it with mask &= mask - 1
.mask &= mask - 1
clears the lowest set bit; trailing_zeros
finds its position first. For example, 1011000 & 1010111 = 1010000
. The hot encode path converts each mask into a flat buffer of boundary offsets, collects up to 256 spans, then processes them in a counted loop. This removes much of the data-dependent control flow from boundary discovery and consumption, but it does not remove the branches and dependent work in cache probes, BPE misses or token emission.
On GPT-2/r50k over a 1 GB OpenWebText sample, the mask scanner reached 2,460 to 2,600 MB/s against 983 MB/s for Gigatoken’s scalar reference, an isolated 2.5x to 2.6x improvement.The r50k module notes report this isolated throughput. The mask scanner makes pretoken boundaries cheaper to find, but the encoder must still turn each span into token IDs.
2. Cache the final IDs for each pretoken
After pretokenisation, an encoder must compute the token IDs for each span, normally by running the BPE merge loop. Gigatoken instead memoises the final token-ID sequence for the exact bytes of each ordinary pretoken. For an unseeded pretoken, the first occurrence computes the tokenizer’s normal answer; later occurrences copy the saved IDs. The cache records the answer for one pretoken, not its merge history or the completed tokenisation of a whole request.
Pretokens up to 15 bytes use a custom short-key table, while longer ones use a separate map. The short table is seeded with exact results for vocabulary byte strings from 1 to 15 bytes. A reused tokenizer instance keeps its results across calls and continues warming.
For a fixed tokenizer, the same pretoken bytes always produce the same IDs, so the cache can replay them without changing the result. Seed values use the tokenizer’s rules rather than assuming that a vocabulary entry’s own ID is always the right answer. The 128-bit key contains the complete short byte string and its length, and the table compares that full key after hashing. A hash collision can cause another probe but cannot return another pretoken’s tokens. Differential tests compare the cached path with uncached encoding across the supported tokenizer families.Implementation and correctness checks: tokenizer-aware seeding, key packing and hashing, and full-key cache probes.
On the author’s 1 GB GPT-2/OpenWebText run, the table accumulated about 1.3 million unique short pretokens and served 99.4% of lookups as hits. About 90% of pretoken occurrences emitted one token and 98% emitted no more than two.The cache source records the OpenWebText distribution and layout rationale. In that workload, most spans skip the pair-rank lookups, merge decisions and scratch-state updates described in the next section. A hit only has to find the cached entry and copy its token IDs.
Fetch the key and answer together
Once caching removes the merge algorithm from the common path, the remaining cost is a largely random table lookup. Processors fetch memory in fixed blocks called cache lines. The x86 machines discussed here use 64-byte lines; the Apple M3 Pro in my benchmark uses 128-byte lines. L1 is the smallest and fastest cache near each core; L2 and the shared last-level cache hold more data at higher latency. A load that misses them may leave the core waiting for main memory.
A short pretoken and its length fit in one 128-bit key. A hash chooses an aligned home pair in the open-addressed table. Where available, Gigatoken calculates the hash with CPU checksum instructions; other targets use a portable arithmetic fallback. Each entry is 32 bytes, so both candidates form one 64-byte probe bucket that fits within a hardware cache line on both architectures. Gigatoken loads both keys and inline values together, compares the complete keys, and selects the match in registers. The common probe therefore requests the line containing the bucket instead of fetching metadata and then waiting for a dependent random load of the value.
Collisions probe later buckets. Up to four token IDs live inside an entry; larger answers spill into a separate append-only token buffer. Cache entries locate spilled answers by offset and length, so buffer reallocation does not invalidate the entries.
Gigatoken processes up to 256 pretokens as a group. While it discovers their spans, it asks the CPU to prefetch each future target line into L2. During the probe pass it requests promotion into L1 sixteen entries before use. A prefetch is a hint, not a guarantee. Its purpose is to overlap the memory fetch with independent work so the core is less likely to stall when the probe needs that line.
The common inline emit path also spends a few extra stores to remove control flow. It reserves room for four token IDs and writes all four lanes from the cache entry. The cursor advances only by the true count, so unused lanes are overwritten by the next result or truncated at the end. That avoids a count-dependent ladder of one-token, two-token, three-token and four-token branches.
There is no published cache-on/cache-off A/B, so I would not assign memoisation alone a multiplier. In the campaign’s cold 10 GB GPT-2/OpenWebText benchmark, the combined probe-and-emit design, including staged prefetch, inline four-token values and flat output, was 27.6% faster in the single-threaded path that materialised the token IDs and 6.2% faster on the multithreaded path.The profiling campaign records both A/Bs under their separate controls.
Reduce address-translation work on Linux
The cache works on ordinary memory pages. On Linux, huge pages can reduce the address-translation overhead as a randomly probed table grows. Before a core can load a cache entry, it must translate the program’s virtual address into a physical address. The processor keeps recent translations in a translation lookaside buffer, or TLB. If the translation is absent, a page-table walk must find it first. On Zen, a software prefetch that misses the data TLB may be dropped, weakening the prefetch ladder described above.
A 64 MiB table occupies 16,384 ordinary 4 KiB pages but only 32 pages if it is fully backed by 2 MiB huge pages. Gigatoken therefore aligns a large short-cache allocation to 2 MiB on Linux and calls MADV_HUGEPAGE
before the memory is first touched. If Linux honours the hint, fewer translations have to cover the same table.
In a separate Zen 5 whole-encode comparison, huge pages reduced warm page walks from about 28.6 million to about 2,300 per pass and improved warm throughput by 7.3%.The Zen 5 profile records the page-walk A/B; Linux documents transparent huge pages. That A/B also covered the input and output, so it measures address translation across the path rather than the cache table alone. The allocation hint is Linux-specific and does nothing on macOS.
3. Make pretoken-cache misses cheaper
A pretoken-cache hit skips BPE. On a miss, the encoder must run the merge loop for that pretoken. Each merge changes up to two neighbouring candidates, so the next choice depends on the previous one. Gigatoken keeps that ordering serial while shortening pair-rank lookups and reusing temporary storage.
Some BPE vocabularies assign merged token IDs in merge-priority order, allowing the ID to rank a candidate. Others store merge rank separately from token ID.Merge priority and token ID can differ. Gigatoken detects that case and includes a reversed-ID test. Gigatoken preserves whichever ordering the tokenizer defines. For an ID-as-rank vocabulary that fits its packed representation, pairs whose IDs are both below 2,048 use a dense 16 MiB grid; other pairs use a packed sparse table. Both replace a general map with a shorter chain of dependent memory loads.See the dense and sparse pair-rank layouts.
Short and medium spans use fixed local rank and neighbour arrays with a linear scan. Long spans use reusable index arrays as a linked list plus a minimum heap, a priority queue that returns the lowest rank. On the ID-as-rank path, Gigatoken retains those arrays and the heap capacity between calls, avoiding fresh allocations for each cache miss. Both paths preserve merge priorities and choose the leftmost candidate when ranks tie.
On a 1 GB GPT-2 run on Zen 5, widening the dense grid to 16 MiB improved whole-encoder single-threaded throughput by 2.8% with a cold pretoken cache.The Zen 5 notes record the 16 MiB dense-grid A/B. Once warm, the cache’s 99.4% hit rate starves this path of work, and the gain disappears.The same dense-grid A/B found no warm-cache gain.
4. Parallelise across proven boundaries
Hugging Face Tokenizers and tiktoken parallelise across caller-supplied inputs, so a million-token document remains one item. Gigatoken finds proven boundaries inside that document and assigns its chunks to several workers.Hugging Face parallelises over caller-supplied inputs; Gigatoken's benchmark notes describe its whole-file input.
Gigatoken cuts only at proven pretoken boundaries, which BPE cannot cross. It will not split added or special tokens; if no safe cut exists, the input stays serial. Tests compare the parallel output with the serial token IDs in their original order.See Gigatoken's safe split-point logic and equivalence tests.
Keep coordination outside the token loop
Workers share immutable vocabulary and pair-rank tables but keep private pretoken caches and scratch buffers. That keeps locks and cross-core traffic from shared writes out of the per-pretoken loop.
Workers claim chunks through an atomic counter. Because the chunks are arranged from largest to smallest, the largest are claimed first while smaller tail chunks keep cores busy near the end. Strict ordering prevents a large chunk from starting late and becoming the final straggler.Source: tail-aware chunk sizing and strict work handout.
Copy results while the tail is still encoding
Gigatoken reserves flat output space and uses a commit cursor to copy the ready prefix while later chunks are still encoding. This overlaps result copying and first-write page allocation with useful work. If the reservation is too small, it gathers the completed chunk buffers after encoding.Source: opportunistic output commits and fallback gathering.
In separate A/B comparisons, the 16-thread path was 6.2% faster with strict handout plus parallel gathering and 4.4% faster with opportunistic prefix copying, each against its own control.The campaign records 6.2% for work handout and gathering and 4.4% for prefix copying.
The closest published total comes from the campaign’s final same-session comparison on a cold 10 GB GPT-2/OpenWebText encode. Its documented single-thread path reached 1,039 MB/s, while the 16-thread path reached 8,792 MB/s. That is about 8.5x the wall throughput, equivalent to cutting the 10 GB encode from roughly 9.6 to 1.14 seconds. Each worker runs the scanner, cache and miss paths described above, so parallelism scales the faster core.The 8.5x comparison is whole-path, not a strict token-identical scaling A/B: its checks report slightly different token counts. A token-identical serial ragged path exists, but no timing is published.
Exclusive worker state removes shared-cache locking from the per-pretoken hot path, but it duplicates the cache and the work needed to warm it. In the author’s 16-worker profile, the state slots accumulated about 16 million distinct entries between them, compared with 5.5 million for a single cache. Aggregate multithreaded CPU time was 14.7 seconds against roughly 11 seconds for Gigatoken’s single-thread run, even though wall time fell sharply. The request finished sooner by spending more aggregate CPU work and memory.
Initial short-cache sizing is clamped between roughly 2 and 128 MiB per state slot, depending on the predicted share of the batch. The tables can continue to grow, however. The short cache has no eviction, and the long-key maps and token arenas are append-only. The pool keeps that memory across requests. One user processing several terabytes reports in an open issue that Gigatoken eventually consumed the RAM and swap of a 1 TB server.The multithreaded profile measures duplicated cache work; issue #36 reports unbounded growth in a long-running job.
The wall-time gain therefore comes with a production requirement: long-running workloads with continually changing input need a way to bound or evict retained cache state.
Gigatoken’s 1,000x headline compares its native whole-buffer API with Hugging Face Tokenizers through its Python-facing batch API. Hugging Face’s encoder is itself multithreaded Rust, so this is not Python code against Rust code. The headline also includes the advantages of handing Gigatoken one 11.9 GB byte buffer, letting it find its own split points and avoiding compatibility work at the Python boundary.The README notes that Hugging Face already runs multithreaded Rust and compatibility mode does not reach 1,000x. The benchmark also gives the systems different input shapes.
My service already used a multithreaded Rust tokenizer core, so my benchmark compared two Rust cores rather than a Python API with Gigatoken. As I reported in my first Gigatoken post, the timed million-token, 1,024-segment count path fell from about 159 milliseconds to 4.24 milliseconds, a 37.6x speed-up.
Together, the four mechanisms show how Gigatoken changes the way tokenisation runs on the hardware: classify 64 bytes at once, lay common cache probes out in 64-byte buckets, shorten the BPE miss path and parallelise at safe boundaries. GPU kernels receive this scrutiny as a matter of course. CPU preprocessing should too.
@misc{doubleword-inside-gigatoken,
title = {The AI Industry Forgot How to Optimise a CPU},
author = {Peter Bhabra},
year = {2026},
howpublished = {Doubleword Blog},
url = {https://blog.doubleword.ai/inside-gigatoken},
}