Your LLM Has Never Read a Word: Tokenization Explained for Developers

A developer has published a technical explainer on tokenization, detailing how large language models convert text into integer token sequences rather than processing words directly. The writeup covers subword tokenization schemes, the distinction between tokens and words, and practical implications for context windows and API costs, illustrated with OpenAI's tiktoken library.

You type a sentence into ChatGPT and it looks like normal English. The model doesn't see it that way. In fact, it never sees words at all. Before the model processes anything, your text is converted into a sequence of numbers. That's what the model actually works with. When I first started learning how LLMs work, I kept hearing terms like tokens , context windows , and token limits . I understood them individually, but I didn't really appreciate how much of modern AI engineering revolves around tokens until I started building visual explanations for them. Once you understand tokenization, a lot of things suddenly make sense: Let's start from the beginning. A token is the basic unit of text that a language model processes. People often assume a token is a word, but that's not quite true. In practice, tokens are usually pieces of words. For example, before the model sees: Hello, world it gets transformed into something like: "Hello", ",", " world", " " Those tokens are then mapped to integer IDs: 15496, 11, 995, 0 At that point, the original text is gone. The model only sees those numbers. A few terms are worth knowing: The analogy that helped me most was this: Tokens are to LLMs what bytes are to computers. We interact with files, images, and videos, but computers ultimately operate on bytes. In the same way, we interact with words and sentences, but language models operate on tokens. One subtle but important detail: the tokenizer is a separate preprocessing step. It is not part of the neural network itself. The same tokenizer used during training must also be used during inference. At first, giving every word its own ID sounds reasonable. The problem is that language is messy. Think about words like: play played playing player Should each be stored separately? Then there are: A pure word-based vocabulary quickly becomes enormous. The opposite approach is character-level tokenization: c a t That solves the vocabulary problem, but creates a different one. Sequences become much longer, making training and inference significantly less efficient. Modern tokenizers use a compromise. Most modern models use some form of subword tokenization. Common words often remain intact, while uncommon words are broken into smaller reusable pieces. For example: unbelievable might become: "un", "believ", "able" This allows a relatively small vocabulary to represent almost any text. The exact algorithm varies between models. You'll often hear names like: The details differ, but the underlying idea is the same: reuse smaller pieces instead of storing every possible word. One of the easiest ways to understand tokenization is to experiment with it. Using OpenAI's tiktoken library: python import tiktoken enc = tiktoken.get encoding "cl100k base" text = "The quick brown fox" ids = enc.encode text print ids print enc.decode i for i in ids Now try: You'll quickly notice that token counts vary much more than most people expect. A sentence with the same number of characters can produce very different token counts depending on the content. This is one of the most common misunderstandings around LLMs. When a model advertises a large context window, that number refers to tokens, not words. Everything shares that budget: Once the limit is reached, something has to be removed or the request fails. Modern models support context windows ranging from hundreds of thousands to over a million tokens, but the exact numbers change frequently as new models are released. As a rough rule of thumb: This becomes particularly important when you're working with large documents. A PDF that looks manageable to a human can easily consume a significant portion of the available context. For most engineers, tokenization becomes important when real-world constraints appear. Most AI APIs charge based on token usage. More input tokens increase prompt costs. More output tokens increase generation costs. Small inefficiencies can become surprisingly expensive when requests scale. Longer prompts require more computation before the model can start generating a response. Longer outputs require more generation steps. More tokens generally means more waiting. Long contexts require larger attention caches and more GPU memory. This becomes one of the major engineering challenges behind large-context models. If each request contains fewer tokens, the same infrastructure can handle more requests. That's why production teams often track token usage as closely as API latency. This was probably the biggest surprise for me. When ChatGPT produces a paragraph, it isn't generating the entire paragraph at once. It predicts a single token. Then another. Each new token becomes part of the context for predicting the next one. Conceptually: "The capital of France is" → "Paris" "The capital of France is Paris" → "." "The capital of France is Paris." → " It" "The capital of France is Paris. It" → " is" Streaming makes the process feel instant, but under the hood it's a sequence of individual predictions. That's one reason longer responses take longer to generate. This is where tokenization stops being theory and becomes a practical engineering problem. Suppose you have a 100-page PDF and want an LLM to answer questions about it. You can't simply send the entire document every time. The document may contain tens of thousands of tokens, leaving little room for prompts or responses. This is why RAG systems exist. Instead of sending everything: Once you start thinking in tokens rather than pages or words, these design decisions become much easier to reason about. | Problem | Token-Related Cause | Typical Fix | |---|---|---| | PDF won't fit | Too many tokens in one request | Chunk into token-sized pieces | | API bill too high | Too many tokens per request | Compress prompts, cache, smaller models | | Slow responses | Long output sequences | Stream and cap output length | | Context keeps truncating | Conversation history grows | Summarize older turns | | RAG results don't fit | Retrieved chunks exceed context | Keep top-K within token budget | Not necessarily. Some words are one token. Others may be several. They're related, but not equivalent. Five characters can be one token, while a single emoji may become multiple tokens. The model ultimately operates on numerical representations derived from token IDs. Prompt length affects cost, latency, memory usage, and throughput. The more I learn about LLMs, the more I feel that tokenization is one of the foundational concepts people skip over too quickly. It's not just a preprocessing step. It influences pricing, context limits, performance, retrieval systems, and even the way responses are generated. Once you start looking at AI systems through the lens of tokens, many of the engineering trade-offs become much easier to understand. While building the AI Foundations section on SeeItFlow, I created an animated walkthrough showing the entire flow: Text → Tokens → Token IDs → Embeddings → Transformer → Response Instead of reading about tokenization, you can watch the process happen step by step and see how the same input text gets transformed before reaching the model. https://seeitflow.com/ai/ai-foundations/tokens-explained https://seeitflow.com/ai/ai-foundations/tokens-explained The next logical question is: If token IDs are just numbers, how do they become something that captures meaning? That's where embeddings come in. I'll cover that in the next article. I built an interactive visualization of this entire process on SeeItFlow , where you can watch text become tokens, tokens become IDs, and responses get generated step by step. What was the moment tokenization finally clicked for you? Let me know in the comments.