{"slug": "gigatoken-rb-a-ruby-port-of-gigatoken-that-s-1-6x-faster", "title": "Gigatoken-rb – A Ruby port of gigatoken that's 1.6x faster", "summary": "Gigatoken-rb, a Ruby port of the gigatoken BPE tokenizer, achieves 12 GB/s (2.78 billion tokens per second) on a Mac Studio M4 Max, running 1.6x faster than the upstream Python package on the same Rust engine. The gem is 340x faster than the fastest existing Ruby gem (tiktoken_ruby) and 1,050x faster than the tokenizers gem, according to benchmarks by developer Jetpks.", "body_md": "**12 GB/s / 2.8 billion tokens per second in Ruby.**\n\nRuby bindings for [marcelroed/gigatoken](https://github.com/marcelroed/gigatoken), the fastest open-source BPE tokenizer around — running **1.6x faster than upstream's own Python package**, on the same Rust engine.\n\n| Corpus | MB/s (median) | Gtok/s (median) | |\n|---|---|---|---|\ngigatoken-rb (this gem, Ruby) |\n11.9 GB | 12,278 |\n2.78 |\n| gigatoken (Python wheel, upstream) | 11.9 GB | 7,400 | 1.68 |\n| tiktoken (Python) | 1.35 GB | 69.7 | 0.0158 |\n| tiktoken_ruby | 1.35 GB | 30.7 | 0.0070 |\n| tokenizers gem (ankane) | 1.35 GB | 10.0 | 0.0023 |\n| tokenizers (Python, Hugging Face) | 1.35 GB | 5.6 | 0.0013 |\n\nMac Studio M4 Max, OpenWebText, GPT-2 tokenizer; every library produces the same tokenization. **340x faster** than the fastest existing Ruby gem (tiktoken_ruby) and **1,050x faster** than the tokenizers gem. Full methodology, exact counts, and the caveats that matter: [docs/rb/benchmarks.md](/jetpks/gigatoken-rb/blob/main/docs/rb/benchmarks.md).\n\n```\ngem install gigatoken\n```\n\nPrecompiled native gems ship for Apple Silicon macOS (`arm64-darwin`\n\n) and x86_64/aarch64 Linux — on those platforms RubyGems grabs the binary automatically, no Rust toolchain, no compile wait. In a Bundler project it's one command:\n\n```\nbundle add gigatoken\n```\n\n(or drop `gem \"gigatoken\"`\n\ninto the Gemfile yourself).\n\nAnywhere else (or with `--platform ruby`\n\nto opt out of the binary), the extension builds from source. That needs a Rust toolchain: `rust-toolchain.toml`\n\npins the nightly, and `rustup`\n\nfetches it automatically on first build.\n\n```\nrequire \"gigatoken\"\n\ntok = Gigatoken::Tokenizer.load(\"openai-community/gpt2\")\n\ntok.encode(\"Hello, world!\")             # => [15496, 11, 995, 0]\ntok.decode([15496, 11, 995, 0])         # => \"Hello, world!\"\ntok.encode_batch([\"Hello!\", \"Another\"]) # => [[15496, 0], [6610]]\n\ntok.vocab_size                          # => 50257\ntok.special_tokens                      # => {\"<|endoftext|>\" => 50256}\n```\n\n`load`\n\ntakes a `tokenizer.json`\n\npath, a directory holding one, a HuggingFace Hub repo id, or a `.tiktoken`\n\nmergeable-ranks file, and dispatches on shape. Hub downloads run over socketry's `async-http`\n\n— no Python anywhere. Know what you have? Skip the dispatch:\n\n```\nGigatoken::Tokenizer.from_file(\"tokenizer.json\")\nGigatoken::Tokenizer.from_hub(\"openai-community/gpt2\", revision: \"main\")\nGigatoken::Tokenizer.from_tiktoken(\"vocab.tiktoken\")\nGigatoken::Tokenizer.from_json(File.binread(\"tokenizer.json\"))\n```\n\nSentencePiece-BPE models (Llama, Gemma, Mistral — any `tokenizer.json`\n\nwith `byte_fallback: true`\n\n) load through the same entry points and pick the right backend automatically. One difference: the SentencePiece core decodes text, so it validates input and raises `Gigatoken::Error`\n\non invalid UTF-8 instead of guessing.\n\n`encode_files`\n\nreads and tokenizes files entirely on the native side — document contents never materialize as Ruby objects. `.gz`\n\nand `.zst`\n\ndecompress transparently.\n\n```\ntok.encode_files(\"owt_train.txt\", separator: \"<|endoftext|>\")\n\njsonl   = Gigatoken::Native::JsonlFileSource.new([\"docs.jsonl\"], field: \"text\")\nparquet = Gigatoken::Native::ParquetFileSource.new([\"docs.parquet\"], column: \"text\")\ntok.encode_files(jsonl)\n```\n\nPass `packed: true`\n\nto `encode_batch`\n\nor `encode_files`\n\nand results land in a single `IO::Buffer`\n\nof u32 token ids instead of a ragged Array of Arrays — no per-token Ruby allocation, the fastest way out of the engine:\n\n```\npacked = tok.encode_files(\"owt_train.txt\", packed: true, separator: \"<|endoftext|>\")\n\npacked.buffer       # => one IO::Buffer, every document's ids back to back\npacked.lens         # => [12, 8, 41, ...] tokens per document\npacked.token_count  # => total tokens\npacked[3]           # => document 3's ids as an Array, on demand\n```\n\n`encode_batch`\n\nand `encode_files`\n\nrelease the GVL for the whole encode; the parallelism runs on the engine's rayon pool, not Ruby threads. Under `Async`\n\n, give the fiber scheduler a worker pool (`ASYNC_SCHEDULER_WORKER_POOL=true`\n\n) and the calling fiber yields to the reactor too. Design notes: [docs/rb/async.md](/jetpks/gigatoken-rb/blob/main/docs/rb/async.md).\n\n```\ngigatoken bench openai-community/gpt2 owt_train.txt --doc-separator \"<|endoftext|>\"\ngigatoken validate openai-community/gpt2 owt_train.txt --doc-separator \"<|endoftext|>\"\n```\n\n`bench`\n\nreports MB/s and Mtok/s (`--packed`\n\nfor the fused packed path, `--no-parallel`\n\nfor the serial core). `validate`\n\nconfirms native split-and-encode agrees with a Ruby-side split through `encode_batch`\n\n.\n\n```\nbundle install\nbundle exec rake compile    # native extension (Rust nightly, via rust-toolchain.toml)\nbundle exec rspec\nbundle exec standardrb\n```\n\nThe Ruby layer is fiber-first throughout — no `Thread`\n\n, no `Mutex`\n\n; all parallelism lives in the core's rayon pool. CI runs ubuntu + macos × Ruby 3.3/3.4/4.0, and `release.yml`\n\ncross-builds the precompiled native gems (arm64-darwin, x86_64-linux, aarch64-linux).\n\nThis fork exists because I need fast tokenization in Ruby. The Rust core is changed as little as possible from upstream. Most of the python shell has been removed from this fork, but you can still find it [upstream](https://github.com/marcelroed/gigatoken).\n\nNot ported/no current plans:\n\n- the HF/tiktoken Python compat shims\n- padded-batch matrices\n- and BPE training\n\nSentencePiece works but — matching upstream — is less optimized than the BPE path.\n\nThe engine is Marcel Rød's gigatoken. If it shows up in your research, cite that:\n\n```\n@software{roed2026gigatoken,\n  author = {Marcel R{\\o}d},\n  title = {{G}igatoken: SIMD and Cache Hierarchies for 1000x Faster Byte-Pair Encoding Tokenization on Modern CPUs},\n  url = {https://github.com/marcelroed/gigatoken},\n  year = {2026},\n}\n```\n\n## AI Use Disclosure\n\nThe Rust engine is upstream's — see [upstream's AI-use disclosure](https://github.com/marcelroed/gigatoken#readme) for how that was built (majority hand-crafted, AI-assisted toward the end).\n\nThe Ruby port in this fork is 100% AI generated using Fable 5 and Sonnet 5 via [space-architect](https://github.com/jetpks/space-architect) over ~24 hours.", "url": "https://wpnews.pro/news/gigatoken-rb-a-ruby-port-of-gigatoken-that-s-1-6x-faster", "canonical_source": "https://github.com/jetpks/gigatoken-rb", "published_at": "2026-07-24 03:32:38+00:00", "updated_at": "2026-07-24 03:52:26.617004+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "natural-language-processing"], "entities": ["gigatoken-rb", "gigatoken", "Jetpks", "Hugging Face", "OpenAI", "Mac Studio M4 Max"], "alternates": {"html": "https://wpnews.pro/news/gigatoken-rb-a-ruby-port-of-gigatoken-that-s-1-6x-faster", "markdown": "https://wpnews.pro/news/gigatoken-rb-a-ruby-port-of-gigatoken-that-s-1-6x-faster.md", "text": "https://wpnews.pro/news/gigatoken-rb-a-ruby-port-of-gigatoken-that-s-1-6x-faster.txt", "jsonld": "https://wpnews.pro/news/gigatoken-rb-a-ruby-port-of-gigatoken-that-s-1-6x-faster.jsonld"}}