Train A BPE Tokenizer

A developer known as soasme released train_tokenizer, a command-line tool that trains and evaluates a byte-level BPE tokenizer using Hugging Face's tokenizers library, with 100 rows of sample data and a test file. The tool demonstrates how byte-pair encoding merges common byte pairs into tokens, balancing vocabulary size and context efficiency, and supports special tokens for production use.

Train A BPE Tokenizer I built train tokenizer https://gist.github.com/soasme/604d4ed9d339d53e5a632a8d5e905654 : a CLI that trains and evaluates a byte-level BPE tokenizer. It has 100 rows of sample data and a test file. What a tokenizer does A language model does not read text. It reads a list of numbers. A tokenizer turns text into that list, and turns the list back into text. The simplest tokenizer splits text on spaces, into words, and gives each word a number. This breaks fast. Any word the model has not seen has no number. A model that never saw “photosynthesizing” cannot represent it. The other extreme splits text into single bytes. This never breaks, since every possible text is a sequence of known bytes. But it wastes space. The word “the” appears in nearly every English sentence. Spending three token slots on it, every time, shrinks how much real content fits in the model’s context window. Byte-pair encoding, or BPE, sits between these two. It builds a vocabulary of chunks bigger than a byte and smaller than a full word. Common byte pairs merge into single tokens. Rare ones stay split. How BPE builds its vocabulary BPE starts from individual bytes. Every unique byte in the training data becomes a token. Then it repeats one step. Find the most common pair of tokens that sit next to each other. Merge that pair into a new token. Take four words: low , lower , newest , widest . Split each into bytes, with a marker for the end of the word: l o w · l o w e r · n e w e s t · w i d e s t · Count every pair of tokens that sit next to each other, across all four words. Each word appears once, so no pair can appear more than twice. Several pairs tie at 2. This example always merges the first tied pair it counted: l o , from low . Merge it into one token, lo . Now low reads lo w · , and lower reads lo w e r · . Repeat. lo w is next, also tied at 2. Merge it into low . Then e s , tied at 2 in newest and widest , merges into es . Then es t merges into est . Then est · is the only pair left at the top, and it merges into est· . Here is that same sequence, played out on all four words: Stop after a fixed number of merges. low and est· are now single tokens. lower keeps e , r , and · as separate pieces, since none of its remaining pairs ever reached the top of the count. Byte-level, not character-level train tokenizer.py starts from raw UTF-8 bytes, not Unicode characters. A byte-level vocabulary starts with the same 256 tokens no matter what language comes in. It needs no per-script setup, and no <unk token for an unseen character. The trade-off appears in non-Latin scripts. A Japanese or Korean character can take three UTF-8 bytes. Byte-level BPE must merge those bytes, one pair at a time, before one character becomes a single token. English mostly needs one byte per letter, so it reaches full-word tokens faster, with fewer merges spent along the way. The code and setup The file splits into four parts: special-token setup, data loading, training, and evaluation. It calls Hugging Face’s tokenizers library for the BPE model and trainer, not a hand-rolled counting loop like the one in the section above. That loop above exists to show the mechanism. The real library does the same counting in Rust, with a priority queue that skips a full recount after every merge, and it is the same code path production training pipelines use. Point this script at a real corpus and a larger vocabulary size, and it trains a tokenizer you could use for real. python def build special tokens special tokens num: int, extra special tokens: Sequence str = , - tuple list str , list str : """Return all special tokens, core special tokens ; only the latter is marked special.""" core = list CORE SPECIAL TOKENS + list extra special tokens fixed = core + STRUCTURAL TOKENS if special tokens num < len fixed : raise ValueError f"special tokens num {special tokens num} is smaller than the " f"number of fixed special tokens {len fixed } " num buffer = special tokens num - len fixed buffer tokens = f"<|buffer{i}| " for i in range 1, num buffer + 1 return fixed + buffer tokens, core <|im start| and <|im end| mark chat turns. <tool call and <think mark structured content. They stay reserved in the vocabulary, but not flagged special, so a model can still emit them as plain text. The buffer tokens extend the total to a fixed number. You can add a new special token later without retraining the tokenizer from scratch. Data loading is one function, read texts : python def read texts data path: str, fmt: str = "text", text field: str = "text", conversations field: str = "conversations", content field: str = "content", max lines: int | None = None, - Iterator str : """Yield training text from a JSONL file. fmt="text": yields record text field . fmt="conversations": newline-joins record conversations field content field . """ if fmt not in "text", "conversations" : raise ValueError f"Unknown format: {fmt r} expected 'text' or 'conversations' " with open data path, "r", encoding="utf-8", errors="ignore" as f: for i, line in enumerate f : if max lines is not None and i = max lines: break line = line.strip if not line: continue try: record = json.loads line except json.JSONDecodeError: continue if fmt == "text": text = record.get text field if text: yield text else: turns = record.get conversations field, contents = t.get content field for t in turns if t.get content field if contents: yield "\n".join contents yield makes this a generator, not a list. It reads one line, parses it, hands the text to the caller, then moves on. It never holds the whole file in memory, only the current line. The training call below pulls from this generator the same way, one item at a time. A JSONL file with a billion rows works the same as one with a hundred: the process still holds only one line at a time. Training itself calls the library’s trainer directly: tokenizer = Tokenizer models.BPE tokenizer.pre tokenizer = pre tokenizers.ByteLevel add prefix space=False tokenizer.decoder = decoders.ByteLevel trainer = trainers.BpeTrainer vocab size=config.vocab size, show progress=config.show progress, initial alphabet=pre tokenizers.ByteLevel.alphabet , special tokens=all special tokens, texts = read texts config.data path, fmt=config.fmt, ... tokenizer.train from iterator texts, trainer=trainer sample data.jsonl holds 100 rows: ten sentences each in English, Spanish, French, German, Russian, Japanese, Korean, Arabic, Hindi, and Greek. Each row is one JSON object, {"text": "...", "lang": "en"} . Running it end to end: pip install -r requirements.txt python train tokenizer.py train --data sample data.jsonl --output-dir ./tokenizer --vocab-size 2000 python train tokenizer.py eval --tokenizer-dir ./tokenizer Results eval applies a chat template, checks that decoding the result gives back the same text, and reports characters per token for each sample language. At a 2,000-token vocabulary: Compression ratio by language chars / tokens : english | ratio: 2.08 spanish | ratio: 2.02 french | ratio: 2.12 german | ratio: 1.81 russian | ratio: 1.36 japanese | ratio: 0.53 korean | ratio: 0.67 arabic | ratio: 1.38 hindi | ratio: 1.24 greek | ratio: 1.31 Average compression ratio: 1.45 English lands at 2.08 characters per token: each token stores about two letters. That is fine for a 2,000-token vocabulary trained on 100 sentences. Production tokenizers train on billions of characters, use 32,000 to 150,000+ tokens, and reach 4-5 characters per token in English. With only 100 rows to learn from, the training data does not have enough repeated pairs to reach whole-word tokens for most of the vocabulary. Japanese and Korean sit under 1.0. Each character there costs more than one token. That is the byte-level trade-off from the section above. A 3-byte character needs several byte merges before it becomes one token. A 2,000-token vocabulary built mostly from Latin-script text does not leave much room for those merges. Full code: gist.github.com/soasme/train tokenizer.py https://gist.github.com/soasme/604d4ed9d339d53e5a632a8d5e905654 .