Building an LLM Inference Engine from Scratch: Tokenization Pipeline Notes

A developer building an LLM inference engine from scratch detailed the tokenization pipeline, which converts raw text into token IDs through normalization, regex pre-tokenization, byte-to-Unicode mapping, and BPE merging. The implementation, using C++ and nlohmann::json, parses Hugging Face tokenizer files and applies iterative merge rules before vocabulary lookup. The project is available on GitHub.

The development of an LLM inference engine starts from understanding and implementing the tokenizer. Before the model can perform any computation, raw text input must be converted into numerical token IDs that correspond to entries in the model vocabulary. The first step was to load the tokenizer files downloaded from Hugging Face. The tokenizer configuration contains several important components: vocab.json : Maps token strings to integer token IDs. merges.txt or equivalent JSON structure : Defines the Byte Pair Encoding BPE merge priority. tokenizer.json : Defines how raw text is initially split into smaller components.Using the nlohmann::json library in C++, the vocabulary and merge dictionaries were parsed into native C++ data structures for efficient lookup. Example structures: std::unordered map<std::string, int vocabulary; std::unordered map< std::pair<std::string, std::string , int, PairHash merge rank; The tokenizer class was designed to manage the complete encoding pipeline: Raw Text | v Normalization | v Regex Pre-tokenization | v Byte / Unicode Conversion | v BPE Merge Algorithm | v Vocabulary Lookup | v Token IDs Modern LLM tokenizers do not directly map words to vocabulary entries. Instead, they apply a multi-stage transformation. The regex pattern stored in tokenizer.json is used to perform pre-tokenization. For example, the pattern separates: A simplified example: Input: Hello world Regex output: "Hello", " world", " " Each segment is then processed independently by the BPE algorithm. Modern LLM tokenizers, such as GPT-2, Qwen, and many Hugging Face models, operate on bytes rather than directly on Unicode characters. The original text: 你好 is first converted into UTF-8 bytes: E4 BD A0 E5 A5 BD Each byte is mapped into a special Unicode representation through a byte-to-unicode mapping. The purpose of this mapping is to allow every possible byte value 0-255 to be represented as a valid Unicode token candidate. Example: Byte: 0xF0 Mapped Unicode: ð This creates an intermediate representation used by BPE. The key discovery during implementation was that tokenization is not simply a vocabulary lookup. The tokenizer does not immediately search: "hello" inside the vocabulary. Instead, it performs iterative merging based on the merge rules. Each pre-tokenized segment is first broken into individual byte/unicode units: Example: hello becomes: h e l l o The tokenizer then checks adjacent pairs: h,e e,l l,l l,o Each pair is searched in the merge dictionary. The merge dictionary contains the priority ranking: php "h","e" - 10 "he","l" - 5 "hel","l" - 3 A lower rank means a higher merge priority. The algorithm repeatedly: After BPE merging is complete, the resulting token strings are searched in the vocabulary dictionary. Example: After merging: "hello", "Ġworld" Vocabulary lookup: php hello - 15339 Ġworld - 1917 The final tokenizer output becomes: 15339, 1917 These integer IDs are then used as input embeddings for the transformer model. Repo: https://github.com/NgKaiWen7/InferenceEngine/tree/tokenization https://github.com/NgKaiWen7/InferenceEngine/tree/tokenization