What is a context window, actually?

A developer explains that an LLM's context window is the total number of tokens it can process at once, and that exceeding this limit causes older information to be forgotten. The post also covers retrieval-augmented generation (RAG) as a technique to provide relevant information without overloading the context window, noting that larger windows are not always better due to cost and attention issues.

AI is moving fast, and it feels like there's a new concept to learn every week. In an effort to actually understand this whole new world instead of just skimming past it, I've been writing ELI5 articles breaking down concepts that show up constantly in AI conversations but rarely get explained simply. This time, a term that gets thrown around a lot without much explanation: the context window. The context window is the total number of input and output tokens an LLM can consider while generating a response. Your prompt, the conversation history, and even the model's response all share that same "budget" of tokens. As a conversation with an LLM grows longer, more tokens live in this window. So far so good. Every AI model has a limit on how many tokens it can hold in its "working memory" at once. So the interesting part with the context window is what happens when you reach these limits. There are several reasons models have context window limits. Processing more tokens requires more memory and computation, making every request slower and more expensive. On top of that, today's models struggle to use very long contexts effectively. They naturally pay more attention to the beginning and end of a conversation than the middle "lost in the middle" problem . An AI model has no persistent memory between conversations. Within a long conversation, it only sees whatever still fits inside its context window. Think of it as a sliding window: as new messages come in, older ones eventually slide out and are no longer visible to the model. This is why a long conversation can feel like the model has "forgotten" something you told it early on. It doesn't actually forget, it just no longer has access to that part of the conversation, unless the product you're using has built something extra on top to handle it. RAG retrieval-augmented generation sounds technical, but the idea is simple: look it up before you answer, instead of guessing from memory. Think of the model as a smart intern who read a huge pile of general knowledge, but has never seen your company's internal docs. If you ask that intern a question about your product, you wouldn't expect them to know the answer off the top of their head. The flow looks more or less like this: That's RAG. Before generating a response, the system retrieves the most relevant pieces of information and adds only those to the prompt. Instead of searching through thousands of documents itself, the model receives just the information it needs. This fixes two problems at once: It's not magic, though. If the search step grabs the wrong page the model's answer will be wrong too. A RAG system is only as good as its retrieval step, not just the model doing the writing at the end. Newer models advertise huge context windows, sometimes hundreds of thousands of tokens. It's tempting to think the fix for all of this is simple: just make the window bigger, and pile everything in. Turns out that doesn't hold up as well as it sounds. It costs more and takes longer. Every extra token in the prompt means more processing on every single request. Models don't "read" a huge context evenly. They tend to pay attention to the start and the end of a long conversation, and quietly lose track of things in the middle. More text means more noise. A huge pile of context gives the model more chances to get distracted by something irrelevant, outdated, or conflicting. A bigger context window isn't automatically better. It gives the model more information to work with, but also more opportunities to be distracted. That's why RAG matters: retrieving a small set of relevant information is usually more effective than hoping the model can make sense of everything at once. Context windows RAG