The context window is a container. Memory is what you decide to put in it. Almost every "memory" conversation conflates the two — here is how they actually work, and how to build a memory layer that does not bankrupt you.
Two months ago I was debugging a customer-support agent for a retail client, and the numbers on the dashboard made no sense. Conversations that should have cost a few cents were running to $0.27 per turn. When I opened the logs, I found the cause in one line: the last message in a 40-turn conversation carried a context of 61,000 tokens, and about 55,000 of those tokens were the entire raw transcript of the conversation, sent back verbatim on every single turn.
The developer who built it had done what every tutorial implies is fine: "the model has a big context window, so just send the whole history." The window was 128k. Nothing crashed. Every turn was technically within budget. But at roughly $0.50 per million input tokens on the model they used, 55,000 redundant tokens per turn, across a support queue handling 1,200 conversations a day, meant they were burning money on tokens that the model had already read and that did not help answer the current question.
That is the gap this article is about. A context window is a capacity. Memory is a policy. The window says how much text the model can look at. Memory is the discipline of deciding what belongs in that window, when, and for how long. The moment you stop conflating the two is the moment your agent stops costing ten times what it should.
Let me be precise about the mechanics, because "context window" gets thrown around like it is a magical vault.
The context window is the set of tokens the model attends to when generating the next token. It is bounded by the model's architecture — the maximum sequence length it was trained to handle — and it is currently anywhere from 4,000 tokens (small models) to 200,000 tokens (flagship models), with a handful of research models pushing toward a million.
Three properties of the window matter in production:
Memory is not one thing. Every serious LLM application uses at least three kinds, and most articles only talk about one. Here is the working taxonomy I use:
This is the raw material the model reasons over right now: the current user message, the last few turns, the retrieved documents, the tool results, the system prompt. Working memory is the context window content. The design job is curation: what subset of history earns a seat.
The brutal constraint is that working memory is where cost and forgetting collide. The more you include, the more you pay and the more the model loses the thread; the less you include, the more context you lose. Every memory system below exists to make this trade-off explicit and cheap.
Episodic memory is the record of what happened in past conversations: the full turns, timestamps, decisions, outcomes. This is the data you need for analytics and for reconstructing a conversation, but it is not what you dump into the window. The retail agent I debugged was shipping episodic memory straight into working memory, uncurated. That is the classic mistake.
The right move is to keep episodic memory in a database — PostgreSQL, Redis, wherever — and only project a curated slice into the window when needed.
Semantic memory is the durable knowledge the agent draws on: product docs, policies, past resolved tickets. This is the vector-database layer. At query time you embed the user's question, retrieve the top-k relevant chunks, and inject them as context. This is what most people mean when they say "memory" in RAG systems, and it is the easiest layer to get wrong because retrieval quality — chunking, embedding model, top-k — determines everything.
Summary memory is the strategy that directly attacks the cost problem: instead of replaying a 55,000-token transcript, you compress it into a running summary — "user is a returning customer, issue is a refund for order #4821, they tried the portal twice, escalated once." Summaries collapse 20 turns into 200 tokens.
The failure mode of summary memory is lossy compression. A summary is a decision about what to forget, and bad summaries forget the detail that matters for the current turn. The professional approach is a tiered design: a short rolling summary for the front of the window, the last few raw turns appended after it (because recent context is usually the most relevant), and the ability to retrieve the full transcript or a specific detail on demand when the summary is not enough.
Here is the concrete mental model I now build against. Imagine the context window as a fixed budget, and assign each category a priority:
┌────────────────────────────────────────────────┐
│ SYSTEM PROMPT (~fixed, always present) │
├────────────────────────────────────────────────┤
│ SUMMARY MEMORY (compressed history) │
├────────────────────────────────────────────────┤
│ LAST 3–5 RAW TURNS (recent context, verbatim)│
├────────────────────────────────────────────────┤
│ RETRIEVED DOCUMENTS (top-k semantic hits) │
├────────────────────────────────────────────────┤
│ CURRENT TOOL OUTPUTS (fresh, task-critical) │
└────────────────────────────────────────────────┘
The system prompt is fixed. The summary is re-computed as the conversation grows. The recent raw turns are the last few, not all. The retrieved documents are the top-k, not the whole knowledge base. The tool outputs are fresh. Every category competes for the same budget, and the ordering above is roughly my priority order when budget runs tight.
The decision rule that drives this: when the window is getting full, compress the past before you drop the present. Never silently truncate recent turns to make room for old ones — users rarely reference turn 30, and they always reference the last thing they said.
Let me make this concrete. This is a minimal but production-shaped summary-memory loop — every N turns, compress the history into a summary, and always keep the last few raw turns for recent context.
import json
from openai import OpenAI
client = OpenAI()
SYSTEM = "You are a customer support agent. Answer using only provided context."
def summarize(history: list[dict]) -> str:
r = client.chat.completions.create(
model="your-summarizer-model",
messages=[
{"role": "system",
"content": "Compress this conversation into a compact running "
"summary: customer identity, issue, actions taken, "
"outstanding decisions. Max 180 words."},
*history,
],
)
return r.choices[0].message.content
class ConversationMemory:
def __init__(self, summary: str = "", raw_limit: int = 4, summarize_every: int = 8):
self.summary = summary
self.raw = [] # recent raw turns (episodic slice)
self.full = [] # full transcript, kept out of the window
self.raw_limit = raw_limit
self.summarize_every = summarize_every
def add(self, role: str, content: str) -> None:
self.full.append({"role": role, "content": content})
self.raw.append({"role": role, "content": content})
if len(self.raw) > self.raw_limit:
self.raw.pop(0)
def maybe_compress(self) -> None:
if len(self.full) >= self.summarize_every:
self.summary = summarize(self.full)
self.full.clear() # transcript archived elsewhere
def window(self, retrieved: list[str]) -> list[dict]:
messages = [{"role": "system", "content": SYSTEM}]
if self.summary:
messages.append({"role": "system",
"content": f"Conversation so far: {self.summary}"})
for chunk in retrieved:
messages.append({"role": "system",
"content": f"Context: {chunk}"})
return messages + self.raw
The shape is what matters, not the library. full
is episodic memory — kept for audit, never dumped into the window wholesale. summary
is the compressed view. raw
is the last four turns, verbatim, because recent context is disproportionately important. retrieved
is the semantic layer, injected as context. When a user asks about something from 20 turns ago, the summary carries the outline and the retrieval layer fills in the detail — and the token bill for a 40-turn conversation drops from 55,000 to a few thousand.
I want to give you the numbers I actually see, because "memory" articles love architecture diagrams and hate invoices.
The cost of not doing this. In that retail agent, the fix cut input tokens per turn by roughly 90% — from 55,000 to around 5,000. At their model's input price, that turned a $0.27-per-turn conversation into a $0.03 one. Across 1,200 conversations a day, that is a saving of roughly $280 a day, or more than $8,000 a month, from one architecture change. Latency improved too, because the model was no longer chewing through 50k redundant tokens before answering.
Where summary memory breaks. Summaries are lossy, and the losses are not random — they skew toward details the summarizer judged "obvious" at the time, which are exactly the details a later turn might need. The fix is a retrieval fallback: when the agent detects it cannot answer from summary-plus-recent, it should query the archived transcript for the specific detail, not guess. I have also seen summary drift — where early wrong facts get baked into the summary and propagated, because the summarizer repeats its own prior summary instead of re-reading the transcript. Re-summarizing from a window of raw turns, rather than from a previous summary, reduces this.
Where retrieval memory breaks. Semantic memory fails quietly: a bad embedding model or aggressive top-k cuts can inject irrelevant context, and the model will happily answer from it. The rule from my RAG work applies double inside a memory system: treat every retrieved chunk as a hypothesis, not a fact, and measure retrieval quality with a labeled evaluation set — never trust the demo.
The context-window ceiling. Even with perfect memory management, some tasks legitimately exceed the window: a 300-page legal contract being analyzed clause by clause, a code review across a large repository. When the working set is bigger than the window and cannot be compressed, the honest answer is chunked, map-reduce-style processing — summarize sections, then reason over the section summaries — or a model with a genuinely larger window, accepting the cost that comes with it.
Not every app needs this machinery, and I have been guilty of over-engineering it. The honest guidance:
The tell I look for: if you are spending more than a few cents per turn on an assistant that answers from history, you have already hit the threshold where a memory policy pays for itself. Measure first. A $30 experiment (log token counts per turn, model the cost) will tell you in an afternoon whether the machinery is worth it.
When you design memory for an LLM application, go through this list:
The developer who built that retail agent was not careless. He was following the loudest advice in the ecosystem — "the window is huge, so send everything" — and the window was indeed huge enough that nothing crashed. That is the trap. Nothing crashing is not the same as something working well. The context window forgives the architecture; the invoice does not.
When I explained the fix to the client, I put it in one sentence: the window is the room, and memory is the furniture. You get to decide what is in the room, and you should never leave 55,000 tokens of clutter in it. That single decision is the difference between an agent that costs $0.27 a turn and one that costs $0.03 — and between an agent that forgets the middle of the conversation and one that remembers exactly what matters, on every single turn.
*Gulshan Yad