{"slug": "how-llms-turn-words-into-numbers-tokenization-and-embeddings-a-simple-guide", "title": "How LLMs Turn Words Into Numbers: Tokenization and Embeddings-A Simple Guide", "summary": "Tokenization converts text into byte sequences that are assigned integer token IDs before entering a transformer model, according to a technical guide on LLM text processing. The guide explains that Byte Pair Encoding (BPE) builds a model's vocabulary by repeatedly merging the most frequent adjacent byte pairs, and that each token ID then maps to a learned embedding vector of 768 to 2048+ dimensions in a real LLM such as Claude.", "body_md": "When you type “I love ice cream” into an LLM, the first thing that happens is **tokenization**. The model doesn’t understand English words directly — it converts your sentence into smaller chunks called **tokens**.\n\n```\nInput: \"I love ice cream\"↓Tokens: [\"I\", \"love\", \"ice\", \"cream\"]\n```\n\nTokens can be whole words, parts of words, or even single characters, depending on the vocabulary the model was trained with.\n\nBefore a model can tokenize anything, it needs a **vocabulary** — a fixed list of all possible tokens it knows. This vocabulary is built using an algorithm called **Byte Pair Encoding (BPE)**.\n\nHere’s how BPE works:\n\nLet’s say we only have three words: low, lower, lowest.\n\nFirst, we break everything into individual characters:\n\n```\nl, o, wL, O, W, E, RL, O, W, E, S, T\n```\n\nWe scan through and find which two characters appear next to each other most often. Let’s say lo appears 3 times.\n\n```\nlo, wlo, w, E, Rlo, w, E, S, T\n```\n\nNow lo is in our vocabulary.\n\nWe find the next most frequent pair. Let’s say low:\n\n```\nlowlow, E, Rlow, E, S, T\n```\n\nWe repeat this process until our vocabulary reaches the desired size. Each merge becomes a new token in our vocabulary.\n\n**Final vocabulary:**\n\n```\n['l', 'o', 'w', 'e', 'r', 's', 't', 'lo', 'low', 'lowe', 'lower', 'lowes', 'lowest']\n```\n\nThis is why BPE is so powerful — it creates a compact vocabulary that captures both common characters and frequently occurring word pieces. It balances vocabulary size with the ability to represent any word.\n\nHere’s a crucial detail: **tokens aren’t directly assigned IDs. They’re first converted to bytes.**\n\nWhy? Because BPE needs to work at the byte level to handle any language, any special character, and any encoding issue uniformly. Bytes are universal — whether your input is English, Chinese, emoji, or binary data, everything converts to bytes the same way.\n\nFor our example:\n\n```\n\"I\"      → bytes → [73]\"love\"   → bytes → [108, 111, 118, 101]\"ice\"    → bytes → [105, 99, 101]\"cream\"  → bytes → [99, 114, 101, 97, 109]\n```\n\nNow BPE operates on these byte sequences, merging the most frequent byte pairs together. After all the merges, our vocabulary contains byte sequences (not raw words):\n\n```\nVocabulary (after BPE):['73', '108-111', '118-101', '105-99-101', '99-114-101-97-109', ...]\n```\n\n(Each item is a sequence of bytes that got merged together.)\n\nNow every byte sequence in our vocabulary gets a unique integer ID:\n\n```\nVocabulary:{  [73]:                    12,  [108, 111]:              2131,  [105, 99, 101]:          56,  [99, 114, 101, 97, 109]: 56456}\n```\n\nSo the ID 12 doesn't point to the word \"I\" — it points to the byte sequence [73], which *represents* \"I\". The ID 2131 points to [108, 111], which is the byte sequence for \"lo\" (a merged pair), part of the word \"love\".\n\nWhen we tokenize “I love ice cream”, we break it into its byte representation, find which merged byte sequences match, look up each one’s ID, and replace it:\n\n```\nInput: \"I love ice cream\"↓Bytes: [73, 108, 111, 118, 101, 105, 99, 101, 99, 114, 101, 97, 109]↓Byte sequences (after matching vocab): [73], [108, 111], [118, 101], [105, 99, 101], [99, 114, 101, 97, 109]↓Token IDs: [12, 2131, 56, 56456]\n```\n\n**This vector of integers is what actually gets sent to the transformer model.**\n\nThe transformer has no idea what “love” means — it only sees 2131, which maps to a byte sequence. The magic is in what happens next.\n\nHere’s where it gets interesting. Each token ID maps to a learned **vector** called an **embedding**. This is different from the vocabulary ID — the embedding is a list of floating-point numbers that capture the *meaning* of the token.\n\nIn a real LLM (like Claude), each token embedding has **768 to 2048+ dimensions**. To keep things simple, let’s imagine each token has a 3-dimensional embedding:\n\n```\nDimension 1: \"Is this a pronoun?\"Dimension 2: \"Is this a thing/noun?\"Dimension 3: \"Is this an action/verb?\"\n```\n\nThese dimensions aren’t explicitly defined — the model **learns** them during training. Here’s what the embeddings for our sentence might look like:\n\n```\nToken: \"I\"Embedding: [1.0, 0.2, 0.1]           (high on pronoun, low on thing, low on action)\nToken: \"love\"Embedding: [0.2, 0.1, 0.9]           (low on pronoun, low on thing, high on action)\nToken: \"ice\"Embedding: [0.1, 0.95, 0.3]           (low on pronoun, high on thing, low on action)\nToken: \"cream\"Embedding: [0.1, 0.95, 0.3]           (low on pronoun, high on thing, low on action)\n```\n\nWhen we plot these in 3D space (imagine a 3D graph where each axis is a learned dimension), we see that tokens with similar meanings cluster together:\n\nThis is the core insight: **embeddings map tokens to meaningful locations in a high-dimensional space.**\n\nNow that “I love ice cream” has become:\n\n```\n[12] → [1.0, 0.2, 0.1][2131] → [0.2, 0.1, 0.9][56] → [0.1, 0.95, 0.3][56456] → [0.1, 0.95, 0.3]\n```\n\nThese embedding vectors are ready to be fed into the transformer. The transformer’s **attention mechanism** (which we’ll explore next) will look at these vectors and figure out how the tokens relate to each other.\n\nWhen attention sees [0.2, 0.1, 0.9] (love) next to [0.1, 0.95, 0.3] (ice cream), it can learn: \"actions often come before things — this is about *doing something to* an object.\"\n\nThe jump from “ice cream” (text) → bytes → 56, 56456 (IDs) → [0.1, 0.95, 0.3] (embeddings) is where raw text becomes something the model can actually reason about.\n\n**Next in this series:** How attention mechanisms find relationships between embeddings.\n\n[How LLMs Turn Words Into Numbers: Tokenization and Embeddings-A Simple Guide](https://pub.towardsai.net/how-llms-turn-words-into-numbers-tokenization-and-embeddings-a-simple-guide-6efd5eb4c216) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/how-llms-turn-words-into-numbers-tokenization-and-embeddings-a-simple-guide", "canonical_source": "https://pub.towardsai.net/how-llms-turn-words-into-numbers-tokenization-and-embeddings-a-simple-guide-6efd5eb4c216?source=rss----98111c9905da---4", "published_at": "2026-09-23 13:01:05+00:00", "updated_at": "2026-09-23 13:30:13.094456+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "artificial-intelligence"], "entities": ["Claude", "Byte Pair Encoding"], "alternates": {"html": "https://wpnews.pro/news/how-llms-turn-words-into-numbers-tokenization-and-embeddings-a-simple-guide", "markdown": "https://wpnews.pro/news/how-llms-turn-words-into-numbers-tokenization-and-embeddings-a-simple-guide.md", "text": "https://wpnews.pro/news/how-llms-turn-words-into-numbers-tokenization-and-embeddings-a-simple-guide.txt", "jsonld": "https://wpnews.pro/news/how-llms-turn-words-into-numbers-tokenization-and-embeddings-a-simple-guide.jsonld"}}