{"slug": "building-an-llm-inference-engine-from-scratch-tokenization-pipeline-notes", "title": "Building an LLM Inference Engine from Scratch: Tokenization Pipeline Notes", "summary": "A developer building an LLM inference engine from scratch detailed the tokenization pipeline, which converts raw text into token IDs through normalization, regex pre-tokenization, byte-to-Unicode mapping, and BPE merging. The implementation, using C++ and nlohmann::json, parses Hugging Face tokenizer files and applies iterative merge rules before vocabulary lookup. The project is available on GitHub.", "body_md": "The development of an LLM inference engine starts from understanding and implementing the tokenizer. Before the model can perform any computation, raw text input must be converted into numerical token IDs that correspond to entries in the model vocabulary.\n\nThe first step was to load the tokenizer files downloaded from Hugging Face. The tokenizer configuration contains several important components:\n\n`vocab.json`\n\n): Maps token strings to integer token IDs.`merges.txt`\n\nor equivalent JSON structure): Defines the Byte Pair Encoding (BPE) merge priority.`tokenizer.json`\n\n): Defines how raw text is initially split into smaller components.Using the `nlohmann::json`\n\nlibrary in C++, the vocabulary and merge dictionaries were parsed into native C++ data structures for efficient lookup.\n\nExample structures:\n\n```\nstd::unordered_map<std::string, int> vocabulary;\n\nstd::unordered_map<\n    std::pair<std::string, std::string>,\n    int,\n    PairHash\n> merge_rank;\n```\n\nThe tokenizer class was designed to manage the complete encoding pipeline:\n\n```\nRaw Text\n   |\n   v\nNormalization\n   |\n   v\nRegex Pre-tokenization\n   |\n   v\nByte / Unicode Conversion\n   |\n   v\nBPE Merge Algorithm\n   |\n   v\nVocabulary Lookup\n   |\n   v\nToken IDs\n```\n\nModern LLM tokenizers do not directly map words to vocabulary entries. Instead, they apply a multi-stage transformation.\n\nThe regex pattern stored in `tokenizer.json`\n\nis used to perform pre-tokenization.\n\nFor example, the pattern separates:\n\nA simplified example:\n\nInput:\n\n```\nHello world!\n```\n\nRegex output:\n\n```\n[\"Hello\", \" world\", \"!\"]\n```\n\nEach segment is then processed independently by the BPE algorithm.\n\nModern LLM tokenizers, such as GPT-2, Qwen, and many Hugging Face models, operate on bytes rather than directly on Unicode characters.\n\nThe original text:\n\n```\n你好\n```\n\nis first converted into UTF-8 bytes:\n\n```\nE4 BD A0\nE5 A5 BD\n```\n\nEach byte is mapped into a special Unicode representation through a byte-to-unicode mapping.\n\nThe purpose of this mapping is to allow every possible byte value (0-255) to be represented as a valid Unicode token candidate.\n\nExample:\n\n```\nByte:\n0xF0\n\nMapped Unicode:\nð\n```\n\nThis creates an intermediate representation used by BPE.\n\nThe key discovery during implementation was that tokenization is not simply a vocabulary lookup.\n\nThe tokenizer does not immediately search:\n\n```\n\"hello\"\n```\n\ninside the vocabulary.\n\nInstead, it performs iterative merging based on the merge rules.\n\nEach pre-tokenized segment is first broken into individual byte/unicode units:\n\nExample:\n\n```\nhello\n```\n\nbecomes:\n\n```\nh e l l o\n```\n\nThe tokenizer then checks adjacent pairs:\n\n```\n(h,e)\n(e,l)\n(l,l)\n(l,o)\n```\n\nEach pair is searched in the merge dictionary.\n\nThe merge dictionary contains the priority ranking:\n\n``` php\n(\"h\",\"e\") -> 10\n(\"he\",\"l\") -> 5\n(\"hel\",\"l\") -> 3\n```\n\nA lower rank means a higher merge priority.\n\nThe algorithm repeatedly:\n\nAfter BPE merging is complete, the resulting token strings are searched in the vocabulary dictionary.\n\nExample:\n\nAfter merging:\n\n```\n[\"hello\", \"Ġworld\"]\n```\n\nVocabulary lookup:\n\n``` php\nhello     -> 15339\nĠworld    -> 1917\n```\n\nThe final tokenizer output becomes:\n\n```\n[\n  15339,\n  1917\n]\n```\n\nThese integer IDs are then used as input embeddings for the transformer model.\n\nRepo: [https://github.com/NgKaiWen7/InferenceEngine/tree/tokenization](https://github.com/NgKaiWen7/InferenceEngine/tree/tokenization)", "url": "https://wpnews.pro/news/building-an-llm-inference-engine-from-scratch-tokenization-pipeline-notes", "canonical_source": "https://dev.to/kai-wen-the-parrot/building-an-llm-inference-engine-from-scratch-tokenization-pipeline-notes-gga", "published_at": "2026-08-05 04:30:00+00:00", "updated_at": "2026-08-05 04:43:57.662951+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools"], "entities": ["Hugging Face", "GPT-2", "Qwen", "nlohmann::json", "C++", "GitHub", "NgKaiWen7"], "alternates": {"html": "https://wpnews.pro/news/building-an-llm-inference-engine-from-scratch-tokenization-pipeline-notes", "markdown": "https://wpnews.pro/news/building-an-llm-inference-engine-from-scratch-tokenization-pipeline-notes.md", "text": "https://wpnews.pro/news/building-an-llm-inference-engine-from-scratch-tokenization-pipeline-notes.txt", "jsonld": "https://wpnews.pro/news/building-an-llm-inference-engine-from-scratch-tokenization-pipeline-notes.jsonld"}}