# Building an LLM Inference Engine from Scratch: Tokenization Pipeline Notes

> Source: <https://dev.to/kai-wen-the-parrot/building-an-llm-inference-engine-from-scratch-tokenization-pipeline-notes-gga>
> Published: 2026-08-05 04:30:00+00:00

The development of an LLM inference engine starts from understanding and implementing the tokenizer. Before the model can perform any computation, raw text input must be converted into numerical token IDs that correspond to entries in the model vocabulary.

The first step was to load the tokenizer files downloaded from Hugging Face. The tokenizer configuration contains several important components:

`vocab.json`

): Maps token strings to integer token IDs.`merges.txt`

or equivalent JSON structure): Defines the Byte Pair Encoding (BPE) merge priority.`tokenizer.json`

): Defines how raw text is initially split into smaller components.Using the `nlohmann::json`

library in C++, the vocabulary and merge dictionaries were parsed into native C++ data structures for efficient lookup.

Example structures:

```
std::unordered_map<std::string, int> vocabulary;

std::unordered_map<
    std::pair<std::string, std::string>,
    int,
    PairHash
> merge_rank;
```

The tokenizer class was designed to manage the complete encoding pipeline:

```
Raw Text
   |
   v
Normalization
   |
   v
Regex Pre-tokenization
   |
   v
Byte / Unicode Conversion
   |
   v
BPE Merge Algorithm
   |
   v
Vocabulary Lookup
   |
   v
Token IDs
```

Modern LLM tokenizers do not directly map words to vocabulary entries. Instead, they apply a multi-stage transformation.

The regex pattern stored in `tokenizer.json`

is used to perform pre-tokenization.

For example, the pattern separates:

A simplified example:

Input:

```
Hello world!
```

Regex output:

```
["Hello", " world", "!"]
```

Each segment is then processed independently by the BPE algorithm.

Modern LLM tokenizers, such as GPT-2, Qwen, and many Hugging Face models, operate on bytes rather than directly on Unicode characters.

The original text:

```
你好
```

is first converted into UTF-8 bytes:

```
E4 BD A0
E5 A5 BD
```

Each byte is mapped into a special Unicode representation through a byte-to-unicode mapping.

The purpose of this mapping is to allow every possible byte value (0-255) to be represented as a valid Unicode token candidate.

Example:

```
Byte:
0xF0

Mapped Unicode:
ð
```

This creates an intermediate representation used by BPE.

The key discovery during implementation was that tokenization is not simply a vocabulary lookup.

The tokenizer does not immediately search:

```
"hello"
```

inside the vocabulary.

Instead, it performs iterative merging based on the merge rules.

Each pre-tokenized segment is first broken into individual byte/unicode units:

Example:

```
hello
```

becomes:

```
h e l l o
```

The tokenizer then checks adjacent pairs:

```
(h,e)
(e,l)
(l,l)
(l,o)
```

Each pair is searched in the merge dictionary.

The merge dictionary contains the priority ranking:

``` php
("h","e") -> 10
("he","l") -> 5
("hel","l") -> 3
```

A lower rank means a higher merge priority.

The algorithm repeatedly:

After BPE merging is complete, the resulting token strings are searched in the vocabulary dictionary.

Example:

After merging:

```
["hello", "Ġworld"]
```

Vocabulary lookup:

``` php
hello     -> 15339
Ġworld    -> 1917
```

The final tokenizer output becomes:

```
[
  15339,
  1917
]
```

These integer IDs are then used as input embeddings for the transformer model.

Repo: [https://github.com/NgKaiWen7/InferenceEngine/tree/tokenization](https://github.com/NgKaiWen7/InferenceEngine/tree/tokenization)