Open Source Project of the Day (#85): tiktoken - OpenAI's Blazing-Fast BPE Tokenizer

OpenAI has released tiktoken, an open-source BPE tokenizer library that powers its GPT model family including GPT-3.5, GPT-4, and GPT-4o. The Rust-based tokenizer converts text strings into token ID sequences for language model consumption and back, enabling developers to precisely control token budgets, estimate costs, and manage context windows before making API calls.

"How many tokens does your prompt actually use?" This is article 85 in the Open Source Project of the Day series. Today's project is tiktoken — OpenAI's official tokenizer. Before calling the OpenAI API, almost every developer runs into the same questions: How many tokens will this text consume? Will it exceed the context limit? How do I estimate the cost? The answers all trace back to a single step: tokenization. tiktoken isn't just a "token counter." It's the actual tokenizer used by the GPT model family during both training and inference. Understanding it means understanding what the model truly "sees" as its input. tiktoken is an open-source BPE Byte Pair Encoding tokenizer library released by OpenAI. Its core job is to convert text strings into sequences of token IDs integer arrays for language models to consume — and to reverse that process, converting token sequences back to the original text. This isn't an experimental side project. It's the tokenizer powering GPT-3.5, GPT-4, GPT-4o, and more. When you send text through the API, the model doesn't see your words — it sees the token sequence tiktoken produces. tiktoken does three things: These three operations are foundational in LLM application development: Token budget control before API calls max tokens errors Smart document chunking for RAG Multi-turn conversation window management Precise cost monitoring Fine-tuning data preprocessing pip install tiktoken python import tiktoken Option 1: Get encoding by name recommended for new projects enc = tiktoken.get encoding "o200k base" Option 2: Get encoding by model name auto-matches the correct encoding enc = tiktoken.encoding for model "gpt-4o" Encode: text → list of token IDs tokens = enc.encode "Hello, tiktoken " print tokens 13225, 11, 384, 4963, 0 print len tokens 5 ← this is the token count Decode: token IDs → text text = enc.decode tokens print text "Hello, tiktoken " Lossless round-trip assert enc.decode enc.encode "Any text can be perfectly restored." == "Any text can be perfectly restored." High-performance Rust core GPT2TokenizerFast Lossless reversibility decode encode text == text always holds — no information is lost in the round-trip Universal coverage High compression ratio Subword awareness ing , tion , pre- , helping models generalize across word forms Multiple built-in encodings o200k base GPT-4o , cl100k base GPT-4/GPT-3.5-turbo , and legacy encodings Special token extension <|im start| to adapt the tokenizer for chat formats Educational module educational module visualizes the BPE merging process step by step| Dimension | tiktoken | HuggingFace Tokenizers | SentencePiece | |---|---|---|---| | Speed | ⚡ Fastest Rust core | Fast Rust core | Medium C++ | | OpenAI model alignment | ✅ Exact match | ❌ Approximate | ❌ N/A | | Python API simplicity | ✅ Minimal | Medium | Medium | | Model coverage | OpenAI series | Universal | Universal | | Custom encodings | ✅ Supported | ✅ Supported | ✅ Supported | Why choose tiktoken? BPE Byte Pair Encoding is tiktoken's core algorithm. Understanding its 4 properties tells you exactly what tiktoken can and cannot do. ① Lossless Reversibility Token sequences reconstruct the original text with 100% fidelity: original = "GPT-4o uses the o200k base encoding." assert enc.decode enc.encode original == original Always true ② Open Vocabulary tiktoken starts from individual bytes 256 characters and merges them by frequency. Every Unicode character can be tokenized — including content the model was never trained on: New words, emoji, source code, edge cases — all tokenized without error enc.encode "😀🤖 tiktoken-v99 unknown word xyz" Never throws ③ High Compression Ratio Each token covers roughly 4 bytes, reducing sequence length and the cost of attention computation: text = "The quick brown fox jumps over the lazy dog" tokens = enc.encode text print f"Characters: {len text }, Tokens: {len tokens }" Characters: 43, Tokens: 9 → ~4.8 chars per token ④ Subword Awareness BPE learns morphological patterns, helping models generalize: "encoding" → "encod", "ing" "tokenization" → "token", "ization" The model can infer the meaning of unseen compound words Using the wrong encoding means your token counts won't match what the API actually charges: | Encoding | Models | Vocabulary Size | |---|---|---| o200k base | GPT-4o, GPT-4o-mini | 200,000 | cl100k base | GPT-4, GPT-3.5-turbo, text-embedding-3- | 100,000 | p50k base | text-davinci-003 and older | 50,000 | r50k base | GPT-3 davinci | 50,000 | php import tiktoken def count tokens text: str, model: str = "gpt-4o" - int: """Count tokens for a given model, exactly matching the API's count.""" enc = tiktoken.encoding for model model return len enc.encode text print count tokens "Hello, world " 4 print count tokens "你好，世界！" 6 Chat-format models use special tokens to delimit roles. You can extend an existing encoding to support them: python import tiktoken cl100k base = tiktoken.get encoding "cl100k base" Build a chat-aware encoding with custom special tokens enc = tiktoken.Encoding name="cl100k im", pat str=cl100k base. pat str, mergeable ranks=cl100k base. mergeable ranks, special tokens={ cl100k base. special tokens, "<|im start| ": 100264, "<|im end| ": 100265, } text = "<|im start| user\nWhat is BPE?<|im end| " tokens = enc.encode text, allowed special={"<|im start| ", "<|im end| "} print f"Token count: {len tokens }" The most common use case — trim message history to fit within the context window before sending: python import tiktoken def trim messages to budget messages: list dict , model: str = "gpt-4o", max tokens: int = 8000, - list dict : """ Trim conversation history so the total token count stays under budget. Preserves the system prompt; drops the oldest user/assistant turns first. """ enc = tiktoken.encoding for model model def count msgs: list dict - int: Each message carries ~4 tokens of overhead role marker, separators total = sum 4 + len enc.encode m.get "content", "" for m in msgs return total + 2 2 tokens priming the reply system = m for m in messages if m "role" == "system" others = m for m in messages if m "role" = "system" while count system + others max tokens and others: others.pop 0 return system + others tiktoken achieves its 3-6x speedup through a Python + Rust hybrid architecture : tiktoken/ ├── tiktoken/ │ ├── init .py ← Public Python API │ ├── core.py ← Encoding class │ ├── model.py ← Model name → encoding name mapping │ ├── registry.py ← Encoding registration and caching │ └── educational.py ← Pure-Python BPE for learning purposes │ └── src/ Rust └── lib.rs ← High-performance BPE core exposed via PyO3 Why Rust makes the difference: tiktoken's value extends well beyond counting tokens. It's the translation layer between developers and GPT models — the component that determines what the model actually "sees." Mastering tiktoken means you can precisely control context windows, estimate costs before they hit your bill, and build LLM applications that behave predictably at the boundaries. Its Python + Rust architecture is also a design pattern worth studying: hand the performance-critical inner loop to a systems language, keep the ergonomics and flexibility in a dynamic language. Simple idea, significant payoff. Find more useful knowledge and interesting products on my Homepage Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.