{"slug": "your-llm-has-never-read-a-word-tokenization-explained-for-developers", "title": "Your LLM Has Never Read a Word: Tokenization Explained for Developers", "summary": "A developer has published a technical explainer on tokenization, detailing how large language models convert text into integer token sequences rather than processing words directly. The writeup covers subword tokenization schemes, the distinction between tokens and words, and practical implications for context windows and API costs, illustrated with OpenAI's tiktoken library.", "body_md": "You type a sentence into ChatGPT and it looks like normal English.\n\nThe model doesn't see it that way.\n\nIn fact, it never sees words at all. Before the model processes anything, your text is converted into a sequence of numbers. That's what the model actually works with.\n\nWhen I first started learning how LLMs work, I kept hearing terms like *tokens*, *context windows*, and *token limits*. I understood them individually, but I didn't really appreciate how much of modern AI engineering revolves around tokens until I started building visual explanations for them.\n\nOnce you understand tokenization, a lot of things suddenly make sense:\n\nLet's start from the beginning.\n\nA token is the basic unit of text that a language model processes.\n\nPeople often assume a token is a word, but that's not quite true. In practice, tokens are usually pieces of words.\n\nFor example, before the model sees:\n\n```\nHello, world!\n```\n\nit gets transformed into something like:\n\n```\n[\"Hello\", \",\", \" world\", \"!\"]\n```\n\nThose tokens are then mapped to integer IDs:\n\n```\n[15496, 11, 995, 0]\n```\n\nAt that point, the original text is gone. The model only sees those numbers.\n\nA few terms are worth knowing:\n\nThe analogy that helped me most was this:\n\nTokens are to LLMs what bytes are to computers.\n\nWe interact with files, images, and videos, but computers ultimately operate on bytes. In the same way, we interact with words and sentences, but language models operate on tokens.\n\nOne subtle but important detail: the tokenizer is a separate preprocessing step. It is not part of the neural network itself. The same tokenizer used during training must also be used during inference.\n\nAt first, giving every word its own ID sounds reasonable.\n\nThe problem is that language is messy.\n\nThink about words like:\n\n```\nplay\nplayed\nplaying\nplayer\n```\n\nShould each be stored separately?\n\nThen there are:\n\nA pure word-based vocabulary quickly becomes enormous.\n\nThe opposite approach is character-level tokenization:\n\n```\nc a t\n```\n\nThat solves the vocabulary problem, but creates a different one. Sequences become much longer, making training and inference significantly less efficient.\n\nModern tokenizers use a compromise.\n\nMost modern models use some form of subword tokenization.\n\nCommon words often remain intact, while uncommon words are broken into smaller reusable pieces.\n\nFor example:\n\n```\nunbelievable\n```\n\nmight become:\n\n```\n[\"un\", \"believ\", \"able\"]\n```\n\nThis allows a relatively small vocabulary to represent almost any text.\n\nThe exact algorithm varies between models. You'll often hear names like:\n\nThe details differ, but the underlying idea is the same: reuse smaller pieces instead of storing every possible word.\n\nOne of the easiest ways to understand tokenization is to experiment with it.\n\nUsing OpenAI's `tiktoken` library:\n\n``` python\nimport tiktoken\n\nenc = tiktoken.get_encoding(\"cl100k_base\")\n\ntext = \"The quick brown fox\"\n\nids = enc.encode(text)\n\nprint(ids)\nprint([enc.decode([i]) for i in ids])\n```\n\nNow try:\n\nYou'll quickly notice that token counts vary much more than most people expect.\n\nA sentence with the same number of characters can produce very different token counts depending on the content.\n\nThis is one of the most common misunderstandings around LLMs.\n\nWhen a model advertises a large context window, that number refers to tokens, not words.\n\nEverything shares that budget:\n\nOnce the limit is reached, something has to be removed or the request fails.\n\nModern models support context windows ranging from hundreds of thousands to over a million tokens, but the exact numbers change frequently as new models are released.\n\nAs a rough rule of thumb:\n\nThis becomes particularly important when you're working with large documents. A PDF that looks manageable to a human can easily consume a significant portion of the available context.\n\nFor most engineers, tokenization becomes important when real-world constraints appear.\n\nMost AI APIs charge based on token usage.\n\nMore input tokens increase prompt costs.\n\nMore output tokens increase generation costs.\n\nSmall inefficiencies can become surprisingly expensive when requests scale.\n\nLonger prompts require more computation before the model can start generating a response.\n\nLonger outputs require more generation steps.\n\nMore tokens generally means more waiting.\n\nLong contexts require larger attention caches and more GPU memory.\n\nThis becomes one of the major engineering challenges behind large-context models.\n\nIf each request contains fewer tokens, the same infrastructure can handle more requests.\n\nThat's why production teams often track token usage as closely as API latency.\n\nThis was probably the biggest surprise for me.\n\nWhen ChatGPT produces a paragraph, it isn't generating the entire paragraph at once.\n\nIt predicts a single token.\n\nThen another.\n\nEach new token becomes part of the context for predicting the next one.\n\nConceptually:\n\n```\n\"The capital of France is\" → \"Paris\"\n\"The capital of France is Paris\" → \".\"\n\"The capital of France is Paris.\" → \" It\"\n\"The capital of France is Paris. It\" → \" is\"\n```\n\nStreaming makes the process feel instant, but under the hood it's a sequence of individual predictions.\n\nThat's one reason longer responses take longer to generate.\n\nThis is where tokenization stops being theory and becomes a practical engineering problem.\n\nSuppose you have a 100-page PDF and want an LLM to answer questions about it.\n\nYou can't simply send the entire document every time.\n\nThe document may contain tens of thousands of tokens, leaving little room for prompts or responses.\n\nThis is why RAG systems exist.\n\nInstead of sending everything:\n\nOnce you start thinking in tokens rather than pages or words, these design decisions become much easier to reason about.\n\n| Problem | Token-Related Cause | Typical Fix | \n|---|---|---|\n| PDF won't fit | Too many tokens in one request | Chunk into token-sized pieces | \n| API bill too high | Too many tokens per request | Compress prompts, cache, smaller models | \n| Slow responses | Long output sequences | Stream and cap output length | \n| Context keeps truncating | Conversation history grows | Summarize older turns | \n| RAG results don't fit | Retrieved chunks exceed context | Keep top-K within token budget | \n\nNot necessarily. Some words are one token. Others may be several.\n\nThey're related, but not equivalent. Five characters can be one token, while a single emoji may become multiple tokens.\n\nThe model ultimately operates on numerical representations derived from token IDs.\n\nPrompt length affects cost, latency, memory usage, and throughput.\n\nThe more I learn about LLMs, the more I feel that tokenization is one of the foundational concepts people skip over too quickly.\n\nIt's not just a preprocessing step.\n\nIt influences pricing, context limits, performance, retrieval systems, and even the way responses are generated.\n\nOnce you start looking at AI systems through the lens of tokens, many of the engineering trade-offs become much easier to understand.\n\nWhile building the AI Foundations section on SeeItFlow, I created an animated walkthrough showing the entire flow:\n\n**Text → Tokens → Token IDs → Embeddings → Transformer → Response**\n\nInstead of reading about tokenization, you can watch the process happen step by step and see how the same input text gets transformed before reaching the model.\n\n[https://seeitflow.com/ai/ai-foundations/tokens-explained](https://seeitflow.com/ai/ai-foundations/tokens-explained)\n\nThe next logical question is:\n\nIf token IDs are just numbers, how do they become something that captures meaning?\n\nThat's where embeddings come in.\n\nI'll cover that in the next article.\n\nI built an interactive visualization of this entire process on **SeeItFlow**, where you can watch text become tokens, tokens become IDs, and responses get generated step by step.\n\nWhat was the moment tokenization finally clicked for you? Let me know in the comments.", "url": "https://wpnews.pro/news/your-llm-has-never-read-a-word-tokenization-explained-for-developers", "canonical_source": "https://dev.to/mangeshmandlik/your-llm-has-never-read-a-word-tokenization-explained-for-developers-4g9l", "published_at": "2026-09-19 09:03:40+00:00", "updated_at": "2026-09-19 09:24:33.112436+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "developer-tools", "ai-tools"], "entities": ["OpenAI", "ChatGPT", "tiktoken"], "alternates": {"html": "https://wpnews.pro/news/your-llm-has-never-read-a-word-tokenization-explained-for-developers", "markdown": "https://wpnews.pro/news/your-llm-has-never-read-a-word-tokenization-explained-for-developers.md", "text": "https://wpnews.pro/news/your-llm-has-never-read-a-word-tokenization-explained-for-developers.txt", "jsonld": "https://wpnews.pro/news/your-llm-has-never-read-a-word-tokenization-explained-for-developers.jsonld"}}