{"slug": "gigatoken-1000x-faster-language-model-tokenization", "title": "GigaToken: ~1000x faster Language model tokenization", "summary": "GigaToken, a new open-source tokenizer, claims to be up to ~1000x faster than HuggingFace's tokenizers and tiktoken for language model tokenization, achieving speeds of over 24 GB/s on an AMD EPYC 9565 processor. The drop-in replacement supports a wide range of CPU hardware and commonly used tokenizers, and can tokenize the entire Common Crawl (130 trillion tokens) in under 6.5 hours according to benchmarks. Developer Niels Rogge demonstrated the tool's performance on Apple M4 Max and AMD EPYC systems, showing speedups of 1353x and 989x respectively over HuggingFace tokenizers.", "body_md": "~1000x faster than HuggingFace's tokenizers, drop-in replacement.\n\n*Tokenize your text data at GB/s!*\n\nNote that both HF tokenizers and tiktoken are already running multithreaded Rust!\n\nGigatoken is the fastest tokenizer for language modeling. It supports a wide range of CPU hardware, and nearly all commonly used tokenizers.\n\n```\npip install gigatoken\n```\n\nGigatoken can be used with its own API, or in compatibility mode with HuggingFace Tokenizers or Tiktoken.\n\n``` python\nimport gigatoken as gt\n\n# Minimum change from existing HuggingFace tokenizers usage (compatibility mode)\nhf_tokenizer = ...\ntokenizer = gt.Tokenizer(hf_tokenizer).as_hf()\n\n# tokenizer can be used in the same contexts as hf_tokenizer\ntokens = tokenizer.encode_batch([\"This is a test string\", \"And here is another\"])\n\n# OR with tiktoken\ntiktokenizer = ...\ntokenizer = gt.Tokenizer(tiktokenizer).as_tiktoken()\n\n# Now works like existing tiktoken tokenizers\ntokens = tokenizer.encode_batch([\"This is a test string\", \"And here is another\"])\n```\n\nA substantial amount of effort has been put into making sure the outputs match exactly with what you would get with HuggingFace Tokenizers in this setting, but this is at a non-negligible cost to performance. You can still expect way faster performance across the board, but not quite the 1000x you will get with the Gigatoken API.\n\n``` python\nimport gigatoken as gt\n\ntokenizer = gt.Tokenizer(\"Qwen/Qwen3-8B\")  # Accepts HF model names\nfile_source = gt.TextFileSource([\"owt_train.txt\"], separator=b\"<|endoftext|>\")\ntokens = tokenizer.encode_files(file_source)\n```\n\nUsing the Gigatoken API lets the Rust implementation read data directly, and skips as much overhead as possible while allowing for maximum parallelism. Keep in mind that passing Python data structures through this API still incurs the overhead of reading from Python.\n\nNo, I way over-optimized for every combination of these! The results are very consistent across CPUs (modern x86 and ARM), and across specific tokenizers.\n\nThe major improvements are in optimizing heavily an implementation that usually is outsourced to a Regex engine (pretokenization) using SIMD and other tricks, as well as heavily optimizing caching of pretoken mappings (if a word has been seen before, look it up its encoded tokens efficiently). In addition, interactions with Python are minimized, and threads are minimally interacting with each other.\n\nYou can try it out without installing anything! The following command will validate and time tokenization for a given HuggingFace model repo:\n\n```\n# Download your data\nwget https://huggingface.co/datasets/stanford-cs336/owt-sample/resolve/main/owt_train.txt.gz  # Just an example!\ngunzip owt_train.txt.gz\nuvx --with tokenizers gigatoken bench 'openai-community/gpt2' owt_train.txt \\\n    --validate --doc-separator \"<|endoftext|>\"\ncpu: Apple M4 Max, 16 cores\ngigatoken:    1.432 s |   11920.51 MB at  8327.05 MB/s |  2701.65 Mtok at 1887.23 Mtok/s\n       hf:   16.250 s |     100.00 MB at     6.15 MB/s |    22.76 Mtok at    1.40 Mtok/s\ngigatoken is 1353.13x faster than hf\nvalidation OK: 20401 documents match\ncpu: AMD EPYC 9565 72-Core Processor, 144 cores, 2 sockets\ngigatoken:    0.486 s |   11920.51 MB at 24532.45 MB/s |  2701.65 Mtok at 5564.94 Mtok/s\n       hf:    4.033 s |     100.00 MB at    24.80 MB/s |    22.76 Mtok at    5.63 Mtok/s\ngigatoken is 989.21x faster than hf\nvalidation OK: 20401 documents match\n```\n\nAt the rates we see on the EPYC CPU, you could tokenize the [entirety of Common Crawl](https://arxiv.org/pdf/2211.04325) (often considered to be the entire internet, 130 trillion tokens) in just under 6.5 hours!\n\nThis example uses the train sample from [this dataset](https://huggingface.co/datasets/stanford-cs336/owt-sample), and the CLI by default subsets to the first 100MB of the file for validation and comparison with HF.\nYou can see help for these flags with `uvx gigatoken bench --help`\n\n.\nYou might need to run your commands twice on macOS to get a good reading, since the first run will always perform a security scan, which will slow down the Rust code.\n\nMost likely not! Despite reasonably wide testing I don't have every use-case on hand, so please report anything you find in a [GitHub Issue](https://github.com/marcelroed/gigatoken/issues) so I can address it as soon as possible.\n\n**Encoding throughput on owt_train.txt (11.9 GB) — AMD EPYC 9565 72-Core Processor x 2 sockets (144 cores)**\n\n| Tokenizer | gigatoken | HF tokenizers | tiktoken | vs HF | vs tiktoken |\n|---|---|---|---|---|---|\n| GPT-2 | 24.53 GB/s | 24.8 MB/s | 36.0 MB/s | 989× | 681× |\n| Phi-4 | 24.00 GB/s | 29.9 MB/s | — | 801× | — |\n| GPT-OSS | 23.96 GB/s | 49.7 MB/s | 42.8 MB/s | 482× | 560× |\n| OLMo 2 / 3 | 23.06 GB/s | 27.7 MB/s | — | 833× | — |\n| Nemotron 3 | 22.79 GB/s | 49.4 MB/s | — | 462× | — |\n| Qwen 3 | 22.16 GB/s | 34.2 MB/s | — | 648× | — |\n| Llama 3 / 3.1 / 3.2 | 22.15 GB/s | 48.5 MB/s | — | 457× | — |\n| GLM 5 | 20.97 GB/s | 74.8 MB/s | — | 280× | — |\n| Llama 3.3 | 20.82 GB/s | 48.3 MB/s | — | 431× | — |\n| Llama 4 | 20.77 GB/s | 72.7 MB/s | — | 286× | — |\n| GLM 4 | 20.61 GB/s | 72.3 MB/s | — | 285× | — |\n| Phi-4-mini | 20.05 GB/s | 27.6 MB/s | — | 726× | — |\n| DeepSeek V3 / R1 / V4 | 19.69 GB/s | 26.2 MB/s | — | 750× | — |\n| Qwen 2 / 2.5 | 19.12 GB/s | 27.7 MB/s | — | 691× | — |\n| Kimi K2 | 18.85 GB/s | — | — | — | — |\n| Qwen 3.5 / 3.6 | 15.49 GB/s | 27.7 MB/s | — | 558× | — |\n| Gemma 4 | 4.82 GB/s | 334.1 MB/s | — | 14× | — |\n| ModernBERT | 4.18 GB/s | 26.9 MB/s | — | 155× | — |\n| Mistral 7B v0.3 | 3.57 GB/s | 354.7 MB/s | — | 10× | — |\n| TinyLlama / Phi-3 (Llama 2) | 3.48 GB/s | 323.6 MB/s | — | 11× | — |\n| CodeLlama | 3.47 GB/s | 347.4 MB/s | — | 10.0× | — |\n| Gemma 3 | 3.43 GB/s | 357.2 MB/s | — | 9.6× | — |\n| Gemma 1 | 2.51 GB/s | 342.2 MB/s | — | 7.3× | — |\n\n**Encoding throughput on owt_train.txt (11.9 GB) — Apple M4 Max (16 cores)**\n\n| Tokenizer | gigatoken | HF tokenizers | tiktoken | vs HF | vs tiktoken |\n|---|---|---|---|---|---|\n| GPT-2 | 8.79 GB/s | 6.9 MB/s | 62.8 MB/s | 1,268× | 140× |\n| Nemotron 3 | 7.82 GB/s | 10.9 MB/s | — | 715× | — |\n| Phi-4 | 7.76 GB/s | 7.7 MB/s | — | 1,012× | — |\n| Llama 3 / 3.1 / 3.2 | 7.60 GB/s | 11.2 MB/s | — | 676× | — |\n| OLMo 2 / 3 | 7.56 GB/s | 5.8 MB/s | — | 1,299× | — |\n| Llama 3.3 | 7.50 GB/s | 15.7 MB/s | — | 479× | — |\n| Phi-4-mini | 6.97 GB/s | 7.2 MB/s | — | 964× | — |\n| Kimi K2 | 6.88 GB/s | — | — | — | — |\n| Llama 4 | 6.81 GB/s | 11.6 MB/s | — | 590× | — |\n| Qwen 2 / 2.5 | 6.37 GB/s | 5.8 MB/s | — | 1,105× | — |\n| Qwen 3 | 6.36 GB/s | 6.9 MB/s | — | 918× | — |\n| Qwen 3.5 / 3.6 | 6.31 GB/s | 6.3 MB/s | — | 994× | — |\n| GPT-OSS | 6.20 GB/s | 20.2 MB/s | 87.2 MB/s | 306× | 71× |\n| GLM 4 | 6.17 GB/s | 15.8 MB/s | — | 392× | — |\n| DeepSeek V3 / R1 / V4 | 5.68 GB/s | 7.2 MB/s | — | 788× | — |\n| GLM 5 | 5.55 GB/s | 12.2 MB/s | — | 456× | — |\n| ModernBERT | 2.64 GB/s | 5.8 MB/s | — | 452× | — |\n| Mistral 7B v0.3 | 1.99 GB/s | 95.1 MB/s | — | 21× | — |\n| Gemma 4 | 1.82 GB/s | 85.2 MB/s | — | 21× | — |\n| CodeLlama | 1.73 GB/s | 80.2 MB/s | — | 22× | — |\n| TinyLlama / Phi-3 (Llama 2) | 1.69 GB/s | 80.1 MB/s | — | 21× | — |\n| Gemma 1 | 1.42 GB/s | 85.7 MB/s | — | 17× | — |\n| Gemma 3 | 1.38 GB/s | 82.2 MB/s | — | 17× | — |\n\n**Encoding throughput on owt_train.txt (11.9 GB) — AMD Ryzen 7 9800X3D 8-Core Processor (16 cores)**\n\n| Tokenizer | gigatoken | HF tokenizers | tiktoken | vs HF | vs tiktoken |\n|---|---|---|---|---|---|\n| GPT-2 | 6.27 GB/s | 59.0 MB/s | 92.1 MB/s | 106× | 68× |\n| Phi-4 | 6.09 GB/s | 55.4 MB/s | — | 110× | — |\n| OLMo 2 / 3 | 6.06 GB/s | 55.4 MB/s | — | 109× | — |\n| Phi-4-mini | 5.80 GB/s | 54.6 MB/s | — | 106× | — |\n| GPT-OSS | 5.68 GB/s | 79.6 MB/s | 112.7 MB/s | 71× | 50× |\n| Qwen 3 | 5.34 GB/s | 54.4 MB/s | — | 98× | — |\n| Qwen 2 / 2.5 | 5.30 GB/s | 51.7 MB/s | — | 103× | — |\n| Llama 3.3 | 5.26 GB/s | 79.9 MB/s | — | 66× | — |\n| Llama 3 / 3.1 / 3.2 | 5.24 GB/s | 79.5 MB/s | — | 66× | — |\n| Kimi K2 | 5.23 GB/s | — | — | — | — |\n| Qwen 3.5 / 3.6 | 5.22 GB/s | 51.6 MB/s | — | 101× | — |\n| Nemotron 3 | 5.20 GB/s | 79.0 MB/s | — | 66× | — |\n| GLM 5 | 5.05 GB/s | 79.5 MB/s | — | 63× | — |\n| GLM 4 | 5.04 GB/s | 79.5 MB/s | — | 63× | — |\n| Llama 4 | 5.03 GB/s | 78.2 MB/s | — | 64× | — |\n| DeepSeek V3 / R1 / V4 | 4.21 GB/s | 51.6 MB/s | — | 82× | — |\n| ModernBERT | 2.84 GB/s | 52.1 MB/s | — | 54× | — |\n| Mistral 7B v0.3 | 1.47 GB/s | 91.6 MB/s | — | 16× | — |\n| Gemma 4 | 1.45 GB/s | 78.8 MB/s | — | 18× | — |\n| CodeLlama | 1.38 GB/s | 85.2 MB/s | — | 16× | — |\n| TinyLlama / Phi-3 (Llama 2) | 1.37 GB/s | 84.9 MB/s | — | 16× | — |\n| Gemma 1 | 1.14 GB/s | 84.9 MB/s | — | 13× | — |\n| Gemma 3 | 1.12 GB/s | 83.0 MB/s | — | 13× | — |\n\n**Benchmark details**\n\nOWT (openwebtext) was chosen because it's roughly representative of the text you get after extraction from CommonCrawl documents.\nGigatoken encodes the whole file un-split, and is thus doing more work than the other tokenizers to find the split boundaries and automatically parallelize.\nHuggingFace tokenizers (`encode_batch_fast`\n\n) gets the first 100 MB and tiktoken (`encode_ordinary_batch`\n\n) the first 1 GB, both presplit on `<|endoftext|>`\n\n.\nThis is fair because neither of the compared tokenizers do caching, meaning the speed is roughly uniform throughout processing.\nTiktoken rows are currently only filled in for tokenizers with official support.\n\nThe slowest rows are the SentencePiece-based tokenizers, which are not well optimized in Gigatoken.\n\nEach row is one distinct tokenizer (identical vocab/merges/pretokenizer), measured on a representative repo. If you don't see your tokenizer here, it's likely based on some existing one. For instance:\n\n**Llama 3 / 3.1 / 3.2**— Llama 3 / 3.1 / 3.2, DeepSeek-R1-Distill-Llama, Hermes 3, Saiga, and other Llama-3 finetunes** Llama 3.3**— Llama 3.3, Llama-3.1-Nemotron-Nano-VL, SmolLM3, Kanana 1.5, jina-embeddings-v5, Ultravox** Qwen 2 / 2.5**— Qwen 2 and 2.5 (incl. Coder and VL), Qwen3-Coder, Qwen3-VL, DeepSeek-R1 Qwen distills, MiMo V2.5, MiniCPM-o 2.6, InternVL3**Qwen 3**— Qwen 3 (incl. Embedding and Reranker), Qwen2.5-Omni, Qwen3-VL-Embedding, MiMo V2.5 Pro, jina-reranker-m0, pplx-embed, MOSS-TTS, Zeta**DeepSeek V3 / R1 / V4**— DeepSeek V3 / V3.1 / V3.2, R1, V4 Flash and Pro, DeepSeek-VL2** GLM 4**— GLM 4.1V, 4.5, and 4.7** GLM 5**— GLM 5 / 5.2 and GLM-4.7-Flash** Nemotron 3**— Nemotron 3 Nano, Super, and Ultra** Kimi K2**— Kimi K2 / K2.5 / K2.6 / K2.7, Kimi-Linear, Kimi-VL, Moonlight** Phi-4-mini**— Phi-4-mini and Phi-4-multimodal** TinyLlama / Phi-3 (Llama 2)**— TinyLlama, Phi-3-mini, Phi-3.5-mini and Phi-3.5-vision (the Llama 2 vocab)** Gemma 3**— Gemma 3 (270M–27B) and EmbeddingGemma** Gemma 4**— Gemma 4 (dense, MoE, and E-series) and DiffusionGemma\n\nIf you use Gigatoken in your research, please cite it as:\n\n```\n@software{roed2026gigatoken,\n  author = {Marcel R{\\o}d},\n  title = {{G}igatoken: SIMD and Cache Hierarchies for 1000x Faster Byte-Pair Encoding Tokenization on Modern CPUs},\n  url = {https://github.com/marcelroed/gigatoken},\n  year = {2026},\n}\n```\n\n- Python iteration is handled in Rust, but uses ABI3, which is slower than using internal version-specific CPython APIs. In the future I intend to specialize for each Python version to cut this overhead. Early experiments show a 2x speed improvement for overhead-bound cases.\n- File sinks are not yet implemented in the Gigatoken API.\n- WordPiece is not yet supported.\n- SentencePiece-based tokenization is not nearly as optimized as the more common BPE tokenizers. This is low priority for now since mostly Google models/BERT style models use SentencePiece.\n- Windows has not been tested much, so for now prefer using WSL.\n\n## AI Use Disclosure\n\nA majority of this code base was crafted by hand without any use of AI (which can be seen from the project's Git history). In the final stages of the project, AI was used to assist:- Implementing the user-facing API\n- Widening of compatibility, for instance generalizing and porting the pretokenizer implementations to support more tokenizers, less interesting features like padding/truncation/unicode normalization\n- Porting SIMD strategies between AVX512/AVX2/NEON\n- Final profiling stages and the last ~4x worth of performance from eliminating branching and improving the pretoken cache hierarchy\n- Refactoring and code reuse", "url": "https://wpnews.pro/news/gigatoken-1000x-faster-language-model-tokenization", "canonical_source": "https://github.com/marcelroed/gigatoken/", "published_at": "2026-07-22 17:20:38+00:00", "updated_at": "2026-07-22 17:36:18.496532+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools"], "entities": ["GigaToken", "HuggingFace", "tiktoken", "OpenAI", "AMD EPYC 9565", "Apple M4 Max", "Common Crawl", "Niels Rogge"], "alternates": {"html": "https://wpnews.pro/news/gigatoken-1000x-faster-language-model-tokenization", "markdown": "https://wpnews.pro/news/gigatoken-1000x-faster-language-model-tokenization.md", "text": "https://wpnews.pro/news/gigatoken-1000x-faster-language-model-tokenization.txt", "jsonld": "https://wpnews.pro/news/gigatoken-1000x-faster-language-model-tokenization.jsonld"}}