# Open Source Project of the Day (#85): tiktoken - OpenAI's Blazing-Fast BPE Tokenizer > Source: > Published: 2026-06-04 01:51:06+00:00 "How many tokens does your prompt actually use?" This is article **#85** in the *Open Source Project of the Day* series. Today's project is **tiktoken** — OpenAI's official tokenizer. Before calling the OpenAI API, almost every developer runs into the same questions: How many tokens will this text consume? Will it exceed the context limit? How do I estimate the cost? The answers all trace back to a single step: tokenization. tiktoken isn't just a "token counter." It's the actual tokenizer used by the GPT model family during both training and inference. Understanding it means understanding what the model truly "sees" as its input. tiktoken is an open-source BPE (Byte Pair Encoding) tokenizer library released by OpenAI. Its core job is to convert text strings into sequences of token IDs (integer arrays) for language models to consume — and to reverse that process, converting token sequences back to the original text. This isn't an experimental side project. It's the tokenizer powering GPT-3.5, GPT-4, GPT-4o, and more. When you send text through the API, the model doesn't see your words — it sees the token sequence tiktoken produces. tiktoken does three things: These three operations are foundational in LLM application development: **Token budget control before API calls** `max_tokens` errors**Smart document chunking for RAG** **Multi-turn conversation window management** **Precise cost monitoring** **Fine-tuning data preprocessing** ``` pip install tiktoken python import tiktoken # Option 1: Get encoding by name (recommended for new projects) enc = tiktoken.get_encoding("o200k_base") # Option 2: Get encoding by model name (auto-matches the correct encoding) enc = tiktoken.encoding_for_model("gpt-4o") # Encode: text → list of token IDs tokens = enc.encode("Hello, tiktoken!") print(tokens) # [13225, 11, 384, 4963, 0] print(len(tokens)) # 5 ← this is the token count # Decode: token IDs → text text = enc.decode(tokens) print(text) # "Hello, tiktoken!" # Lossless round-trip assert enc.decode(enc.encode("Any text can be perfectly restored.")) == "Any text can be perfectly restored." ``` **High-performance Rust core** `GPT2TokenizerFast` )**Lossless reversibility** `decode(encode(text)) == text` always holds — no information is lost in the round-trip**Universal coverage** **High compression ratio** **Subword awareness** `ing` , `tion` , `pre-` ), helping models generalize across word forms**Multiple built-in encodings** `o200k_base` (GPT-4o), `cl100k_base` (GPT-4/GPT-3.5-turbo), and legacy encodings**Special token extension** `<|im_start|>` to adapt the tokenizer for chat formats**Educational module** `_educational` module visualizes the BPE merging process step by step| Dimension | tiktoken | HuggingFace Tokenizers | SentencePiece | |---|---|---|---| | Speed | ⚡ Fastest (Rust core) | Fast (Rust core) | Medium (C++) | | OpenAI model alignment | ✅ Exact match | ❌ Approximate | ❌ N/A | | Python API simplicity | ✅ Minimal | Medium | Medium | | Model coverage | OpenAI series | Universal | Universal | | Custom encodings | ✅ Supported | ✅ Supported | ✅ Supported | **Why choose tiktoken?** BPE (Byte Pair Encoding) is tiktoken's core algorithm. Understanding its 4 properties tells you exactly what tiktoken can and cannot do. **① Lossless Reversibility** Token sequences reconstruct the original text with 100% fidelity: ``` original = "GPT-4o uses the o200k_base encoding." assert enc.decode(enc.encode(original)) == original # Always true ``` **② Open Vocabulary** tiktoken starts from individual bytes (256 characters) and merges them by frequency. Every Unicode character can be tokenized — including content the model was never trained on: ``` # New words, emoji, source code, edge cases — all tokenized without error enc.encode("😀🤖 tiktoken-v99 unknown_word_xyz") # Never throws ``` **③ High Compression Ratio** Each token covers roughly 4 bytes, reducing sequence length and the cost of attention computation: ``` text = "The quick brown fox jumps over the lazy dog" tokens = enc.encode(text) print(f"Characters: {len(text)}, Tokens: {len(tokens)}") # Characters: 43, Tokens: 9 → ~4.8 chars per token ``` **④ Subword Awareness** BPE learns morphological patterns, helping models generalize: ``` # "encoding" → ["encod", "ing"] # "tokenization" → ["token", "ization"] # The model can infer the meaning of unseen compound words ``` Using the wrong encoding means your token counts won't match what the API actually charges: | Encoding | Models | Vocabulary Size | |---|---|---| `o200k_base` | GPT-4o, GPT-4o-mini | 200,000 | `cl100k_base` | GPT-4, GPT-3.5-turbo, text-embedding-3-* | 100,000 | `p50k_base` | text-davinci-003 and older | 50,000 | `r50k_base` | GPT-3 (davinci) | 50,000 | ``` php import tiktoken def count_tokens(text: str, model: str = "gpt-4o") -> int: """Count tokens for a given model, exactly matching the API's count.""" enc = tiktoken.encoding_for_model(model) return len(enc.encode(text)) print(count_tokens("Hello, world!")) # 4 print(count_tokens("你好，世界！")) # 6 ``` Chat-format models use special tokens to delimit roles. You can extend an existing encoding to support them: ``` python import tiktoken cl100k_base = tiktoken.get_encoding("cl100k_base") # Build a chat-aware encoding with custom special tokens enc = tiktoken.Encoding( name="cl100k_im", pat_str=cl100k_base._pat_str, mergeable_ranks=cl100k_base._mergeable_ranks, special_tokens={ **cl100k_base._special_tokens, "<|im_start|>": 100264, "<|im_end|>": 100265, } ) text = "<|im_start|>user\nWhat is BPE?<|im_end|>" tokens = enc.encode(text, allowed_special={"<|im_start|>", "<|im_end|>"}) print(f"Token count: {len(tokens)}") ``` The most common use case — trim message history to fit within the context window before sending: ``` python import tiktoken def trim_messages_to_budget( messages: list[dict], model: str = "gpt-4o", max_tokens: int = 8000, ) -> list[dict]: """ Trim conversation history so the total token count stays under budget. Preserves the system prompt; drops the oldest user/assistant turns first. """ enc = tiktoken.encoding_for_model(model) def count(msgs: list[dict]) -> int: # Each message carries ~4 tokens of overhead (role marker, separators) total = sum(4 + len(enc.encode(m.get("content", ""))) for m in msgs) return total + 2 # 2 tokens priming the reply system = [m for m in messages if m["role"] == "system"] others = [m for m in messages if m["role"] != "system"] while count(system + others) > max_tokens and others: others.pop(0) return system + others ``` tiktoken achieves its 3-6x speedup through a **Python + Rust hybrid architecture**: ``` tiktoken/ ├── tiktoken/ │ ├── __init__.py ← Public Python API │ ├── core.py ← Encoding class │ ├── model.py ← Model name → encoding name mapping │ ├── registry.py ← Encoding registration and caching │ └── _educational.py ← Pure-Python BPE for learning purposes │ └── src/ (Rust) └── lib.rs ← High-performance BPE core (exposed via PyO3) ``` **Why Rust makes the difference:** tiktoken's value extends well beyond counting tokens. It's the translation layer between developers and GPT models — the component that determines what the model actually "sees." Mastering tiktoken means you can precisely control context windows, estimate costs before they hit your bill, and build LLM applications that behave predictably at the boundaries. Its Python + Rust architecture is also a design pattern worth studying: hand the performance-critical inner loop to a systems language, keep the ergonomics and flexibility in a dynamic language. Simple idea, significant payoff. *Find more useful knowledge and interesting products on my Homepage* *Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.*