The development of an LLM inference engine starts from understanding and implementing the tokenizer. Before the model can perform any computation, raw text input must be converted into numerical token IDs that correspond to entries in the model vocabulary.
The first step was to load the tokenizer files downloaded from Hugging Face. The tokenizer configuration contains several important components:
vocab.json
): Maps token strings to integer token IDs.merges.txt
or equivalent JSON structure): Defines the Byte Pair Encoding (BPE) merge priority.tokenizer.json
): Defines how raw text is initially split into smaller components.Using the nlohmann::json
library in C++, the vocabulary and merge dictionaries were parsed into native C++ data structures for efficient lookup.
Example structures:
std::unordered_map<std::string, int> vocabulary;
std::unordered_map<
std::pair<std::string, std::string>,
int,
PairHash
> merge_rank;
The tokenizer class was designed to manage the complete encoding pipeline:
Raw Text
|
v
Normalization
|
v
Regex Pre-tokenization
|
v
Byte / Unicode Conversion
|
v
BPE Merge Algorithm
|
v
Vocabulary Lookup
|
v
Token IDs
Modern LLM tokenizers do not directly map words to vocabulary entries. Instead, they apply a multi-stage transformation.
The regex pattern stored in tokenizer.json
is used to perform pre-tokenization.
For example, the pattern separates:
A simplified example:
Input:
Hello world!
Regex output:
["Hello", " world", "!"]
Each segment is then processed independently by the BPE algorithm.
Modern LLM tokenizers, such as GPT-2, Qwen, and many Hugging Face models, operate on bytes rather than directly on Unicode characters.
The original text:
你好
is first converted into UTF-8 bytes:
E4 BD A0
E5 A5 BD
Each byte is mapped into a special Unicode representation through a byte-to-unicode mapping.
The purpose of this mapping is to allow every possible byte value (0-255) to be represented as a valid Unicode token candidate.
Example:
Byte:
0xF0
Mapped Unicode:
ð
This creates an intermediate representation used by BPE.
The key discovery during implementation was that tokenization is not simply a vocabulary lookup.
The tokenizer does not immediately search:
"hello"
inside the vocabulary.
Instead, it performs iterative merging based on the merge rules.
Each pre-tokenized segment is first broken into individual byte/unicode units:
Example:
hello
becomes:
h e l l o
The tokenizer then checks adjacent pairs:
(h,e)
(e,l)
(l,l)
(l,o)
Each pair is searched in the merge dictionary.
The merge dictionary contains the priority ranking:
("h","e") -> 10
("he","l") -> 5
("hel","l") -> 3
A lower rank means a higher merge priority.
The algorithm repeatedly:
After BPE merging is complete, the resulting token strings are searched in the vocabulary dictionary.
Example:
After merging:
["hello", "Ġworld"]
Vocabulary lookup:
hello -> 15339
Ġworld -> 1917
The final tokenizer output becomes:
[
15339,
1917
]
These integer IDs are then used as input embeddings for the transformer model.
Repo: https://github.com/NgKaiWen7/InferenceEngine/tree/tokenization