Prompt engineering always felt a little ridiculous — but here’s the truth nobody says out loud: C** ontext is King.** Picture this: you have at most a million tokens to tell your LLM everything it needs to know about your business and exactly what you want your agent to accomplish. Sounds like a lot, right?
Now actually add up what has to fit in there — and you’ll realize it’s nowhere near enough space to tackle complex, real-world tasks. So if this is going to work at all, we need to understand the building blocks of context in modern agentic applications, and we need a framework to design that context with precision.
Let’s remember what an LLM even is. It’s a stateless box that takes tokens as input — think of them as little pieces of words — and gives us likely tokens as output. It’s important to come back to this core idea. It’s very smart and does a lot of really cool things. The likely tokens that come out are extremely useful, but it has no persistent memory across independent requests unless memory is implemented outside the model. We give it a bunch of text on the input, and text comes out as the output.
When I said you have about a million tokens, that’s for some models; for others it might be a hundred thousand, or something like that. That could correspond to roughly 60,000 words, or somewhere in the range of 60,000 to 700,000 words.
But here’s the catch with an LLM: its knowledge comes from its training data, and on its own, it can’t actually do anything. It just hands us tokens and stops there. What we’re really after is different. We want to build agentic applications that take real action and actually understand what’s going on inside the business.
For our agents, we’ll have some input — probably a prompt from a human at some point. This agent is going to make LLM calls, probably iteratively; it will call an LLM more than once. It also has the capability of calling tools, which let it actually take action out in the world.
Both the LLM call and the tool call can be iterative. This agent can have a workflow that it works out over time, and it can run for a long time, depending on the agent. That introduces some real problems with context that we need to solve.
Here’s a minimal shape of that loop. The important part isn’t the framework code itself, but the repeated cycle: call the LLM, let it decide whether to finish or use a tool, execute that tool, and feed the result back into the next iteration.
def run_agent(user_message: str, tools: dict, max_steps: int = 10): """A simplified agentic loop around a stateless LLM.""" context = build_initial_context(user_message) for _ in range(max_steps): response = llm_call(context) if response.is_final: return response.content if response.tool_call: # Keep the assistant's tool request in the conversation history. context.append_assistant_message(response) tool = tools.get(response.tool_call.name) if tool is None: context.append_tool_result( response.tool_call, {"error": f"Unknown tool: {response.tool_call.name}"} ) continue # Execute the tool and add its result to the next context. result = tool(**response.tool_call.args) context.append_tool_result(response.tool_call, result) return "Reached step limit without a final answer."
Let’s now look at a schema for thinking about context. What goes into the context window — these tokens, this big prompt, this string we submit to our LLM? There are six things I want to lay out.
It’s fine to think of the chatbot interface — the typical Grok, Claude, or ChatGPT experience. You type a thing, and it says stuff back. The thing you type is the user message, and that’s probably going to be the initial input to our agentic service.
That’s not all. There is always a system prompt. In chatbot applications, you have some capability of customizing it. It’s always present, even if you don’t see it, and you can always tune it. It covers things like the LLM's personality: what do you want it to be? Do you want a harsh and demanding coach, or a gentle and nurturing counsellor?
It gives the model guardrails — like “don’t give me bomb-making instructions,” or “be specific and detailed and cite your sources in your output.” All sorts of things might go in a system prompt. It sits up there so you don’t have to say all that over and over again. It’s always present.
We also have tools we might have access to. The whole point is that the agent has access to tools, and in its terminal step, when it’s done with its work, that’s a tool call causing an effect out in the world. So as part of this context — the text we’re passing into the LLM — we have a description of the tools available to us.
We also have resources. This is a complex topic on its own. Because the LLM only knows what it learned during training and what it reads on the internet, we have to tell it things specific to our business. Hopefully, all your Google Docs are not shared publicly — your wiki articles, internal support tickets, and all that kind of stuff.
There really is private data inside your enterprise that is never going to be part of any LLM’s training dataset. Those are the resources. We have to do a good job of picking which resources belong inside this context window in the submission to the LLM — things relevant to the user message, enough but not too much. We’ll talk more about that in a bit.
You’ve also got the assistant messages. Remember the chat model: you type stuff, it says stuff back — it is your assistant. You give it a user message, and it responds; that is the assistant message. As you iterate, which is the case when you’re using a chatbot application, you accumulate a history of user messages and a history of assistant messages inside the context window. So the longer the agent iterates, the more space this takes up.
Finally, you’ve got tool calls and responses — calls and results. As an agent does its work, it may call tools as intermediate steps, not to terminate execution with its full result, but to do other things that help build up context so it can finish its work. So you may have a history of tool calls and responses as part of this too.
This isn’t a provider-specific API structure; it’s a mental model for thinking about everything that can consume tokens in an agentic application. The important point is that the context isn’t just the user’s prompt — it also includes instructions, tools, resources, history, and interactions with tools.
These three components — the history of tool calls, user messages, and assistant messages — grow as execution unfolds. Resources I’m drawing as the same-size box as the rest, but it might not be the same size; that could be a substantial number of tokens taken up by resources. The system prompt is probably reasonable.
Tools are probably limited. But these other things can take a lot of space. When you start to look at what really goes into context, you begin to understand that even the big models with a million tokens — which, again, is not all of them — don’t feel very roomy anymore when you’re building complex agentic applications.
And it gets a little worse. It’s not always the case that more is better. If you’re using a model with, say, a 100,000-token context window, it’s pretty consistent that maxing it out doesn’t give you your best results. Somewhere in the 60 to 70% range — and let the research unfold on this and let that number stabilize — is probably where you get your best results. So more isn’t always better. We actually need tools for engineering what goes in here.
One practical way to apply this idea is to treat the advertised context limit as a ceiling, not necessarily as your target. The exact threshold is workload-dependent, so this example uses 65% only as a starting point to evaluate rather than a universal rule.
def context_budget( max_tokens: int, target_utilization: float = 0.65) -> int: """ Starting heuristic for the usable context budget. The optimal utilization depends on the model, workload, prompt structure, and evaluation results, so this value should be tuned rather than treated as a universal rule. """ return int(max_tokens * target_utilization)# Example: for a 100k-token context window, start by evaluating# a workload around 65k tokens rather than assuming that using# the entire window will always produce the best results.usable = context_budget(100_000) # 65000
Let’s look at that set of tools. There are four things I want to cover.
First is the system prompt itself. Writing a good prompt is important — we need to get this right. It’s a one-off for our agent, and you want to nail it. There’s a bit of a Goldilocks problem here. In one ditch, we could be too vague: “I don’t know, do a good job and go get all the stuff you need to give me a good answer.” That’s kind of stupid, and it’s not really going to work.
On the other side, we could be too prescriptive. For example, if you find yourself in your system prompt defining if-then logic and saying “make sure you do this, and if this happens, do that” — you want to let the LLM figure that out. That’s what it’s going to be good at, so don’t do that.
You want to be in the middle, where you define outcomes and maybe broad approaches to things. I realize this is itself somewhat vague, but it’s a balance you’re going to have to learn how to strike. That is the art of prompt engineering: making sure you’re not too prescriptive and not too vague. So engineer the system prompt — that’s something you actually have to pay attention to.
Next is tools — pretty simple here. You want to make sure your tool descriptions are specific and detailed. You never want them to be too long; more is not always better. But you want them to really nail down what the tool does, so the LLM can understand what that is for. Be specific, and include a schema. We have to know what goes into that tool and what comes out.
That’s important because the LLM needs to know it has all the right inputs before concluding it can call the tool, and it needs to know what it gets out, because that output may be an input to a next step — say, a resource it needs to get. So use full schemas and specific, precise tool descriptions.
For example, instead of giving the model a vague description like “look up a user,” we can expose a clear contract: what the tool does, what it accepts, and what it returns.
get_user_record = { "name": "get_user_record", "description": ( "Fetch the full profile for a single user by their user ID. " "Use this when you have a user ID but need details such as " "name, plan, and account status." ), "input_schema": { "type": "object", "properties": { "user_id": {"type": "string", "description": "Unique user identifier."} }, "required": ["user_id"], }, "output_schema": { "type": "object", "properties": { "user_id": {"type": "string"}, "name": {"type": "string"}, "plan": {"type": "string"}, "status": {"type": "string"}, }, },}
Next is data retrieval, and this goes right to resources — this is what we’re talking about. The initial approach, when folks first started using LLMs in an enterprise context, was retrieval-augmented generation (RAG). We’ve got a vector database; we index any document we think we might need for a lookup.
When I have a user message, I take it and use it against the vector database to retrieve any documents I think are related, put them into the context, and away I go. That’s good for an information-search, chatbot kind of thing — it worked. It may well still be part of your stack, and I’m not here to tell you that you’re old-fashioned or behind the times. But RAG by itself is not very precise context engineering on a go-forward basis. There’s more we can do to use the context window we’ve got economically.
One increasingly popular approach is MCP (Model Context Protocol), the Model Context Protocol, and its disclosure of resources can be queried. Those resources, like tools, are described with text and maybe with parameters that can be involved in a query if they’re queryable resources. Those descriptions can go into the context window of one call, and a subsequent call, if you ask the LLM, can say, “I need you to go query this resource.”
One of the agent’s multiple iterative calls could simply ask, based on this user message, which resources should I go retrieve? The LLM tells you, based on its conclusion, you get those resources, and now you’ve got a big string. You can inject that into the resources section of your next context window for your next call as the agent moves along.
There are other things we can do to make economical use of this. For instance, if you’ve got a user record that could be long, maybe give it a user ID. If there’s a resource or tool that can clearly turn that user ID into a full record on a subsequent call, you can get the model to ask you — you can prompt it to ask you: if it needs more detail on that user, call this tool or ask for this resource.
So there are all kinds of things we can do to be economical with the data we have in here. Rather than including anything that might sound related, we can describe what’s available and use prompting to have the model tell us what we need to get and include.
The key idea is to describe what resources are available without immediately all of them into the context. Let the model select the resources it needs, then retrieve only those resources for the next call.
Finally, we’ve got long-horizon thinking. This is for an agent we know is going to iterate for a long time. We suspect it might need to make a bunch of calls, and the things that grow as the agent’s runtime increases are going to be our enemy. There are some things we can do to help keep those small, or at least within the context window of any given call.
i) Compaction. One thing LLMs are incredible at is summarizing text. If I’ve got a resource I retrieved on a previous call and it’s 50,000 tokens — a really big thing — I don’t want all of that in my next call. So I could use a single LLM call with my own prompt to say, “Please summarize this in 500 words.” Now I’ve got that compacted summary that goes in as a resource. That can often give you pretty good results.
Compaction is a deliberate trade-off: we spend one LLM call to compress a large resource into a smaller representation that can survive into later steps. The summary is lossy, so the prompt should explicitly preserve facts the agent may need later.
def compact(resource_text: str, word_limit: int = 500) -> str: """ Compress a large resource before carrying it into a later context. The summary is intentionally lossy, so preserve facts, constraints, identifiers, decisions, and numbers that the agent may need later. """ return llm_call( system_prompt=( f"Summarize the following resource in at most {word_limit} words. " "Preserve important facts, constraints, identifiers, decisions, " "and numbers that an agent may need in later steps." ), user_message=resource_text, ).content
ii) Memory. Memory is literally just a little key-value store hanging off to the side. At one point in my agent’s execution, it lets me take some resource, some intermediate result — maybe an assistant message that could itself be lengthy, maybe some structured data like a bunch of JSON or CSV, or who knows what’s going on with this crazy agent you’re writing — and store it under a certain key.
On a subsequent step, when I need it, I retrieve it from that memory. Conceptually, it’s like retrieving resources from an MCP server, but it really is memory local to our agent that we get to use to store that stuff. You don’t have much space in the context window; you might need to put certain things over here, where this space is relatively cheap.
class AgentMemory: """ Minimal in-process store for state kept outside the context window. This is only a conceptual example. A production implementation could use a database, Redis, object storage, or another durable store. """ def __init__(self): self._store: dict[str, object] = {} def put(self, key: str, value: object) -> None: self._store[key] = value def get(self, key: str) -> object | None: return self._store.get(key)memory = AgentMemory()# Keep large intermediate data outside the context window.memory.put("invoice_batch_2026_07", large_json_payload)# later stepbatch = memory.get("invoice_batch_2026_07")
This example is an in-process store; production systems could replace it with durable storage such as a database, cache, or object store.
iii) Additional agents (composition). I can take my one big agent and split it. I think this is just likely to happen as you’re building an agentic application. Tell me if this has ever happened when you’re building a microservice: it starts to get complex, you realize it’s a little bit of its own mini monolith, and you have to split it into pieces. That’s what building software is like. When you’re writing an agent, it’s probably going to end up decomposing into multiple agents.
Let’s say one of your resources is your documentation site or wiki, and you’ve got some complex search logic doing a lot of iteration. That’s just its own agent — so split it off. Now this agent can ask that agent, “I’ve got this text; I’d like to know what documents from your system look relevant.”
Rather than just a text search on that document store, you can have an agent that does thoughtful sorting, prioritizing, and back-and-forth with an LLM to give you a concise summary of what comes from that store. So be ready to take little chunks of agentic functionality in your agent that could really live on their own and make them their own thing.
def docs_search_agent(query: str) -> str: """ A specialized sub-agent that owns document retrieval, ranking, and summarization. """ candidates = vector_search(query, top_k=25) return run_agent( user_message=f"""Rank and summarize the most relevant documents for this request:{query}Candidate documents:{candidates}""", tools={"read_doc": read_doc}, )# The main agent simply delegates document research# to the specialized sub-agent.relevant_docs = docs_search_agent("payroll remittance policy")
This is what composition looks like in practice: instead of making the main agent own every piece of retrieval logic, we give document search its own specialized agent. The main agent can then delegate the research task and receive back a focused result.
So there you go. We’ve got a schema for context: the user message, the system prompt, whatever resources we can gather, a listing of the tools, the assistant messages we get from our back-and-forth with the agent, and of course a record of the tools we’ve called. That’s a lot. It all has to fit into the LLM’s context window, and those aren’t getting a lot bigger.
To manage it, we have these tools: engineer your system prompt, describe your tools well, retrieve data intelligently, use the resources section wisely, and for long-horizon iterative execution, lean on compaction, memory, and composition.
Context engineering is the present and future of agentic AI development. I encourage you to put these tools into practice!
If you found this helpful, consider clapping👏 so others can find it too and follow me for more amazing technical AI content!
Your LLM Has a Million-Token Memory. Here’s Why That’s Still Not Enough. was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.