I built train_tokenizer: a CLI that trains and evaluates a byte-level BPE tokenizer. It has 100 rows of sample data and a test file.
What a tokenizer does
A language model does not read text. It reads a list of numbers. A tokenizer turns text into that list, and turns the list back into text.
The simplest tokenizer splits text on spaces, into words, and gives each word a number. This breaks fast. Any word the model has not seen has no number. A model that never saw “photosynthesizing” cannot represent it.
The other extreme splits text into single bytes. This never breaks, since every possible text is a sequence of known bytes. But it wastes space. The word “the” appears in nearly every English sentence. Spending three token slots on it, every time, shrinks how much real content fits in the model’s context window.
Byte-pair encoding, or BPE, sits between these two. It builds a vocabulary of chunks bigger than a byte and smaller than a full word. Common byte pairs merge into single tokens. Rare ones stay split.
How BPE builds its vocabulary
BPE starts from individual bytes. Every unique byte in the training data becomes a token. Then it repeats one step. Find the most common pair of tokens that sit next to each other. Merge that pair into a new token.
Take four words: low
, lower
, newest
, widest
. Split each into bytes, with a marker for the end of the word:
l o w ·
l o w e r ·
n e w e s t ·
w i d e s t ·
Count every pair of tokens that sit next to each other, across all four words. Each word appears once, so no pair can appear more than twice. Several pairs tie at 2. This example always merges the first tied pair it counted: l o
, from low
. Merge it into one token, lo
. Now low
reads lo w ·
, and lower
reads lo w e r ·
.
Repeat. lo w
is next, also tied at 2. Merge it into low
. Then e s
, tied at 2 in newest
and widest
, merges into es
. Then es t
merges into est
. Then est ·
is the only pair left at the top, and it merges into est·
.
Here is that same sequence, played out on all four words:
Stop after a fixed number of merges. low
and est·
are now single tokens. lower
keeps e
, r
, and ·
as separate pieces, since none of its remaining pairs ever reached the top of the count.
Byte-level, not character-level
train_tokenizer.py
starts from raw UTF-8 bytes, not Unicode characters. A byte-level vocabulary starts with the same 256 tokens no matter what language comes in. It needs no per-script setup, and no <unk>
token for an unseen character.
The trade-off appears in non-Latin scripts. A Japanese or Korean character can take three UTF-8 bytes. Byte-level BPE must merge those bytes, one pair at a time, before one character becomes a single token. English mostly needs one byte per letter, so it reaches full-word tokens faster, with fewer merges spent along the way.
The code and setup
The file splits into four parts: special-token setup, data , training, and evaluation. It calls Hugging Face’s tokenizers
library for the BPE model and trainer, not a hand-rolled counting loop like the one in the section above.
That loop above exists to show the mechanism. The real library does the same counting in Rust, with a priority queue that skips a full recount after every merge, and it is the same code path production training pipelines use. Point this script at a real corpus and a larger vocabulary size, and it trains a tokenizer you could use for real.
def build_special_tokens(
special_tokens_num: int,
extra_special_tokens: Sequence[str] = (),
) -> tuple[list[str], list[str]]:
"""Return (all_special_tokens, core_special_tokens); only the latter is marked special."""
core = list(CORE_SPECIAL_TOKENS) + list(extra_special_tokens)
fixed = core + STRUCTURAL_TOKENS
if special_tokens_num < len(fixed):
raise ValueError(
f"special_tokens_num ({special_tokens_num}) is smaller than the "
f"number of fixed special tokens ({len(fixed)})"
)
num_buffer = special_tokens_num - len(fixed)
buffer_tokens = [f"<|buffer{i}|>" for i in range(1, num_buffer + 1)]
return fixed + buffer_tokens, core
<|im_start|>
and <|im_end|>
mark chat turns. <tool_call>
and <think>
mark structured content. They stay reserved in the vocabulary, but not flagged special, so a model can still emit them as plain text. The buffer tokens extend the total to a fixed number. You can add a new special token later without retraining the tokenizer from scratch.
Data is one function, read_texts
:
def read_texts(
data_path: str,
fmt: str = "text",
text_field: str = "text",
conversations_field: str = "conversations",
content_field: str = "content",
max_lines: int | None = None,
) -> Iterator[str]:
"""Yield training text from a JSONL file.
fmt="text": yields record[text_field].
fmt="conversations": newline-joins record[conversations_field][*][content_field].
"""
if fmt not in ("text", "conversations"):
raise ValueError(f"Unknown format: {fmt!r} (expected 'text' or 'conversations')")
with open(data_path, "r", encoding="utf-8", errors="ignore") as f:
for i, line in enumerate(f):
if max_lines is not None and i >= max_lines:
break
line = line.strip()
if not line:
continue
try:
record = json.loads(line)
except json.JSONDecodeError:
continue
if fmt == "text":
text = record.get(text_field)
if text:
yield text
else:
turns = record.get(conversations_field, [])
contents = [t.get(content_field) for t in turns if t.get(content_field)]
if contents:
yield "\n".join(contents)
yield
makes this a generator, not a list. It reads one line, parses it, hands the text to the caller, then moves on. It never holds the whole file in memory, only the current line. The training call below pulls from this generator the same way, one item at a time. A JSONL file with a billion rows works the same as one with a hundred: the process still holds only one line at a time.
Training itself calls the library’s trainer directly:
tokenizer = Tokenizer(models.BPE())
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
tokenizer.decoder = decoders.ByteLevel()
trainer = trainers.BpeTrainer(
vocab_size=config.vocab_size,
show_progress=config.show_progress,
initial_alphabet=pre_tokenizers.ByteLevel.alphabet(),
special_tokens=all_special_tokens,
)
texts = read_texts(config.data_path, fmt=config.fmt, ...)
tokenizer.train_from_iterator(texts, trainer=trainer)
sample_data.jsonl
holds 100 rows: ten sentences each in English, Spanish, French, German, Russian, Japanese, Korean, Arabic, Hindi, and Greek. Each row is one JSON object, {"text": "...", "lang": "en"}
.
Running it end to end:
pip install -r requirements.txt
python train_tokenizer.py train --data sample_data.jsonl --output-dir ./tokenizer --vocab-size 2000
python train_tokenizer.py eval --tokenizer-dir ./tokenizer
Results
eval
applies a chat template, checks that decoding the result gives back the same text, and reports characters per token for each sample language. At a 2,000-token vocabulary:
Compression ratio by language (chars / tokens):
english | ratio: 2.08
spanish | ratio: 2.02
french | ratio: 2.12
german | ratio: 1.81
russian | ratio: 1.36
japanese | ratio: 0.53
korean | ratio: 0.67
arabic | ratio: 1.38
hindi | ratio: 1.24
greek | ratio: 1.31
Average compression ratio: 1.45
English lands at 2.08 characters per token: each token stores about two letters. That is fine for a 2,000-token vocabulary trained on 100 sentences. Production tokenizers train on billions of characters, use 32,000 to 150,000+ tokens, and reach 4-5 characters per token in English. With only 100 rows to learn from, the training data does not have enough repeated pairs to reach whole-word tokens for most of the vocabulary.
Japanese and Korean sit under 1.0. Each character there costs more than one token. That is the byte-level trade-off from the section above. A 3-byte character needs several byte merges before it becomes one token. A 2,000-token vocabulary built mostly from Latin-script text does not leave much room for those merges.
Full code: gist.github.com/soasme/train_tokenizer.py.