How LLMs Turn Words Into Numbers: Tokenization and Embeddings-A Simple Guide

Tokenization converts text into byte sequences that are assigned integer token IDs before entering a transformer model, according to a technical guide on LLM text processing. The guide explains that Byte Pair Encoding (BPE) builds a model's vocabulary by repeatedly merging the most frequent adjacent byte pairs, and that each token ID then maps to a learned embedding vector of 768 to 2048+ dimensions in a real LLM such as Claude.

When you type “I love ice cream” into an LLM, the first thing that happens is tokenization . The model doesn’t understand English words directly — it converts your sentence into smaller chunks called tokens . Input: "I love ice cream"↓Tokens: "I", "love", "ice", "cream" Tokens can be whole words, parts of words, or even single characters, depending on the vocabulary the model was trained with. Before a model can tokenize anything, it needs a vocabulary — a fixed list of all possible tokens it knows. This vocabulary is built using an algorithm called Byte Pair Encoding BPE . Here’s how BPE works: Let’s say we only have three words: low, lower, lowest. First, we break everything into individual characters: l, o, wL, O, W, E, RL, O, W, E, S, T We scan through and find which two characters appear next to each other most often. Let’s say lo appears 3 times. lo, wlo, w, E, Rlo, w, E, S, T Now lo is in our vocabulary. We find the next most frequent pair. Let’s say low: lowlow, E, Rlow, E, S, T We repeat this process until our vocabulary reaches the desired size. Each merge becomes a new token in our vocabulary. Final vocabulary: 'l', 'o', 'w', 'e', 'r', 's', 't', 'lo', 'low', 'lowe', 'lower', 'lowes', 'lowest' This is why BPE is so powerful — it creates a compact vocabulary that captures both common characters and frequently occurring word pieces. It balances vocabulary size with the ability to represent any word. Here’s a crucial detail: tokens aren’t directly assigned IDs. They’re first converted to bytes. Why? Because BPE needs to work at the byte level to handle any language, any special character, and any encoding issue uniformly. Bytes are universal — whether your input is English, Chinese, emoji, or binary data, everything converts to bytes the same way. For our example: "I" → bytes → 73 "love" → bytes → 108, 111, 118, 101 "ice" → bytes → 105, 99, 101 "cream" → bytes → 99, 114, 101, 97, 109 Now BPE operates on these byte sequences, merging the most frequent byte pairs together. After all the merges, our vocabulary contains byte sequences not raw words : Vocabulary after BPE : '73', '108-111', '118-101', '105-99-101', '99-114-101-97-109', ... Each item is a sequence of bytes that got merged together. Now every byte sequence in our vocabulary gets a unique integer ID: Vocabulary:{ 73 : 12, 108, 111 : 2131, 105, 99, 101 : 56, 99, 114, 101, 97, 109 : 56456} So the ID 12 doesn't point to the word "I" — it points to the byte sequence 73 , which represents "I". The ID 2131 points to 108, 111 , which is the byte sequence for "lo" a merged pair , part of the word "love". When we tokenize “I love ice cream”, we break it into its byte representation, find which merged byte sequences match, look up each one’s ID, and replace it: Input: "I love ice cream"↓Bytes: 73, 108, 111, 118, 101, 105, 99, 101, 99, 114, 101, 97, 109 ↓Byte sequences after matching vocab : 73 , 108, 111 , 118, 101 , 105, 99, 101 , 99, 114, 101, 97, 109 ↓Token IDs: 12, 2131, 56, 56456 This vector of integers is what actually gets sent to the transformer model. The transformer has no idea what “love” means — it only sees 2131, which maps to a byte sequence. The magic is in what happens next. Here’s where it gets interesting. Each token ID maps to a learned vector called an embedding . This is different from the vocabulary ID — the embedding is a list of floating-point numbers that capture the meaning of the token. In a real LLM like Claude , each token embedding has 768 to 2048+ dimensions . To keep things simple, let’s imagine each token has a 3-dimensional embedding: Dimension 1: "Is this a pronoun?"Dimension 2: "Is this a thing/noun?"Dimension 3: "Is this an action/verb?" These dimensions aren’t explicitly defined — the model learns them during training. Here’s what the embeddings for our sentence might look like: Token: "I"Embedding: 1.0, 0.2, 0.1 high on pronoun, low on thing, low on action Token: "love"Embedding: 0.2, 0.1, 0.9 low on pronoun, low on thing, high on action Token: "ice"Embedding: 0.1, 0.95, 0.3 low on pronoun, high on thing, low on action Token: "cream"Embedding: 0.1, 0.95, 0.3 low on pronoun, high on thing, low on action When we plot these in 3D space imagine a 3D graph where each axis is a learned dimension , we see that tokens with similar meanings cluster together: This is the core insight: embeddings map tokens to meaningful locations in a high-dimensional space. Now that “I love ice cream” has become: 12 → 1.0, 0.2, 0.1 2131 → 0.2, 0.1, 0.9 56 → 0.1, 0.95, 0.3 56456 → 0.1, 0.95, 0.3 These embedding vectors are ready to be fed into the transformer. The transformer’s attention mechanism which we’ll explore next will look at these vectors and figure out how the tokens relate to each other. When attention sees 0.2, 0.1, 0.9 love next to 0.1, 0.95, 0.3 ice cream , it can learn: "actions often come before things — this is about doing something to an object." The jump from “ice cream” text → bytes → 56, 56456 IDs → 0.1, 0.95, 0.3 embeddings is where raw text becomes something the model can actually reason about. Next in this series: How attention mechanisms find relationships between embeddings. How LLMs Turn Words Into Numbers: Tokenization and Embeddings-A Simple Guide https://pub.towardsai.net/how-llms-turn-words-into-numbers-tokenization-and-embeddings-a-simple-guide-6efd5eb4c216 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.