Open Source Project of the Day (#85): tiktoken - OpenAI's Blazing-Fast BPE Tokenizer

"How many tokens does your prompt actually use?"

This is article #85 in the Open Source Project of the Day series. Today's project is tiktoken — OpenAI's official tokenizer.

Before calling the OpenAI API, almost every developer runs into the same questions: How many tokens will this text consume? Will it exceed the context limit? How do I estimate the cost? The answers all trace back to a single step: tokenization.

tiktoken isn't just a "token counter." It's the actual tokenizer used by the GPT model family during both training and inference. Understanding it means understanding what the model truly "sees" as its input.

tiktoken is an open-source BPE (Byte Pair Encoding) tokenizer library released by OpenAI. Its core job is to convert text strings into sequences of token IDs (integer arrays) for language models to consume — and to reverse that process, converting token sequences back to the original text.

This isn't an experimental side project. It's the tokenizer powering GPT-3.5, GPT-4, GPT-4o, and more. When you send text through the API, the model doesn't see your words — it sees the token sequence tiktoken produces.

tiktoken does three things:

These three operations are foundational in LLM application development:

Token budget control before API calls

max_tokens

errorsSmart document chunking for RAG

Multi-turn conversation window management

Precise cost monitoring

Fine-tuning data preprocessing

pip install tiktoken
python
import tiktoken

enc = tiktoken.get_encoding("o200k_base")

enc = tiktoken.encoding_for_model("gpt-4o")

tokens = enc.encode("Hello, tiktoken!")
print(tokens)        # [13225, 11, 384, 4963, 0]
print(len(tokens))   # 5  ← this is the token count

text = enc.decode(tokens)
print(text)          # "Hello, tiktoken!"

assert enc.decode(enc.encode("Any text can be perfectly restored.")) == "Any text can be perfectly restored."

High-performance Rust core

GPT2TokenizerFast

)Lossless reversibility

decode(encode(text)) == text

always holds — no information is lost in the round-tripUniversal coverage

High compression ratio

Subword awareness

ing

, tion

, pre-

), helping models generalize across word formsMultiple built-in encodings

o200k_base

(GPT-4o), cl100k_base

(GPT-4/GPT-3.5-turbo), and legacy encodingsSpecial token extension

<|im_start|>

to adapt the tokenizer for chat formatsEducational module

_educational

module visualizes the BPE merging process step by step| Dimension | tiktoken | HuggingFace Tokenizers | SentencePiece | |---|---|---|---| | Speed | ⚡ Fastest (Rust core) | Fast (Rust core) | Medium (C++) | | OpenAI model alignment | ✅ Exact match | ❌ Approximate | ❌ N/A | | Python API simplicity | ✅ Minimal | Medium | Medium | | Model coverage | OpenAI series | Universal | Universal | | Custom encodings | ✅ Supported | ✅ Supported | ✅ Supported |

Why choose tiktoken?

BPE (Byte Pair Encoding) is tiktoken's core algorithm. Understanding its 4 properties tells you exactly what tiktoken can and cannot do.

① Lossless Reversibility

Token sequences reconstruct the original text with 100% fidelity:

original = "GPT-4o uses the o200k_base encoding."
assert enc.decode(enc.encode(original)) == original  # Always true

② Open Vocabulary

tiktoken starts from individual bytes (256 characters) and merges them by frequency. Every Unicode character can be tokenized — including content the model was never trained on:

enc.encode("😀🤖 tiktoken-v99 unknown_word_xyz")  # Never throws

③ High Compression Ratio

Each token covers roughly 4 bytes, reducing sequence length and the cost of attention computation:

text = "The quick brown fox jumps over the lazy dog"
tokens = enc.encode(text)
print(f"Characters: {len(text)}, Tokens: {len(tokens)}")

④ Subword Awareness

BPE learns morphological patterns, helping models generalize:

Using the wrong encoding means your token counts won't match what the API actually charges:

Encoding	Models	Vocabulary Size
o200k_base
GPT-4o, GPT-4o-mini	200,000
cl100k_base
GPT-4, GPT-3.5-turbo, text-embedding-3-*	100,000
p50k_base
text-davinci-003 and older	50,000
r50k_base
GPT-3 (davinci)	50,000

import tiktoken

def count_tokens(text: str, model: str = "gpt-4o") -> int:
    """Count tokens for a given model, exactly matching the API's count."""
    enc = tiktoken.encoding_for_model(model)
    return len(enc.encode(text))

print(count_tokens("Hello, world!"))   # 4
print(count_tokens("你好，世界！"))     # 6

Chat-format models use special tokens to delimit roles. You can extend an existing encoding to support them:

import tiktoken

cl100k_base = tiktoken.get_encoding("cl100k_base")

enc = tiktoken.Encoding(
    name="cl100k_im",
    pat_str=cl100k_base._pat_str,
    mergeable_ranks=cl100k_base._mergeable_ranks,
    special_tokens={
        **cl100k_base._special_tokens,
        "<|im_start|>": 100264,
        "<|im_end|>":   100265,
    }
)

text = "<|im_start|>user\nWhat is BPE?<|im_end|>"
tokens = enc.encode(text, allowed_special={"<|im_start|>", "<|im_end|>"})
print(f"Token count: {len(tokens)}")

The most common use case — trim message history to fit within the context window before sending:

import tiktoken

def trim_messages_to_budget(
    messages: list[dict],
    model: str = "gpt-4o",
    max_tokens: int = 8000,
) -> list[dict]:
    """
    Trim conversation history so the total token count stays under budget.
    Preserves the system prompt; drops the oldest user/assistant turns first.
    """
    enc = tiktoken.encoding_for_model(model)

    def count(msgs: list[dict]) -> int:
        total = sum(4 + len(enc.encode(m.get("content", ""))) for m in msgs)
        return total + 2  # 2 tokens priming the reply

    system = [m for m in messages if m["role"] == "system"]
    others = [m for m in messages if m["role"] != "system"]

    while count(system + others) > max_tokens and others:
        others.pop(0)

    return system + others

tiktoken achieves its 3-6x speedup through a Python + Rust hybrid architecture:

tiktoken/
├── tiktoken/
│   ├── __init__.py       ← Public Python API
│   ├── core.py           ← Encoding class
│   ├── model.py          ← Model name → encoding name mapping
│   ├── registry.py       ← Encoding registration and caching
│   └── _educational.py   ← Pure-Python BPE for learning purposes
│
└── src/ (Rust)
    └── lib.rs            ← High-performance BPE core (exposed via PyO3)

Why Rust makes the difference:

tiktoken's value extends well beyond counting tokens. It's the translation layer between developers and GPT models — the component that determines what the model actually "sees." Mastering tiktoken means you can precisely control context windows, estimate costs before they hit your bill, and build LLM applications that behave predictably at the boundaries.

Its Python + Rust architecture is also a design pattern worth studying: hand the performance-critical inner loop to a systems language, keep the ergonomics and flexibility in a dynamic language. Simple idea, significant payoff.

Find more useful knowledge and interesting products on my Homepage

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

source & further reading

dev.to — original article Why your LLM tool calls silently break — and a ~10µs fix Give Your AI Agent Persistent Memory Without Touching Its Internals I Tried to Build an AI Code Reviewer Without Sharing My Code — Here's What Worked