Context Engineering: Giving LLMs the Right Context

A developer outlines context engineering, the practice of deciding what information an LLM receives for a given query rather than sending the entire conversation history. The writeup contrasts it with prompt engineering and uses a customer support chatbot to show why stuffing full histories breaks down through context-window limits, rising token costs and latency, noise, conflicting and stale information.

Introduction When you chat with ChatGPT, Claude, or Gemini, it feels like you are talking to a bot which remembers everything. You ask a question, the AI understands what you mean, and it can keep the conversation going without making you repeat everything. But how is it remembering what you said earlier? The AI model is given enough information to answer the current query. This information can come from the current conversation, user details, retrieved documents, tool outputs, application state, and several other sources. For example, consider a customer support chatbot. User: Hi. I want to return my keyboard. Chatbot: Sure. Please provide your order number. User: My order number is ORD-772. For the last query, the model may receive something like: Hi. I want to return my keyboard. Sure. Please provide your order number. My order number is ORD-772. The model can now understand that ORD-772 is the order number for the keyboard the customer wants to return. This simple approach works. But what happens when the conversation becomes 20, 50, or 100 messages long? As the conversation grows, continuously sending everything to the model becomes difficult to manage. Why sending everything does not scale There are several issues with this approach. - Context window: Every model has a limit on how much input it can process in a single request. If the entire conversation becomes larger than the available context window, the application cannot send everything to the model. - Increasing token usage: If we keep sending the entire conversation on every request, the number of input tokens keeps increasing. More tokens can mean higher API cost and increased latency. - Too much noise: Not everything in the conversation is useful for the current query. Sending unrelated information can make the context unnecessarily large. - Conflicting information: The conversation may contain multiple versions of the same information, especially when data or the users preferences changes over time. - Stale information: Some information may have changed since it was mentioned earlier in the conversation. So the solution is not bigger context, but what context does the model actually need for the current query? This is where context engineering becomes important. What is Context Engineering? Let’s first understand what context is. Context is the information that is provided to an LLM when it generates a response. It can contain much more than the user’s current message. For a customer support chatbot, the context could contain: - System instructions - Customer information - Current order information - Conversation history - Policy documents - Tool outputs - Current task - User query Context engineering is the process of deciding: - What information should be included? - What information can be removed? - What information should be summarized or compressed? - What information should have higher priority? - How much space should each piece of information get? - How should the information be structured before sending it to the model? The goal is not to provide everything, but to give the right information to the model for the current task Context Engineering vs Prompt Engineering Prompt engineering focuses on how we write instructions for the model to generate the desired response. For example: You are a customer support assistant. Be polite and concise. Do not make up information. Writing good instructions is still important. But real applications require much more than a well-written prompt. In addition to the prompt, it should be able to manage all the other information, like past conversations, tool outputs, external documents, etc. Prompt engineering focuses mainly on how we instruct the model. Context engineering focuses on how we manage the information given to the model. A Customer Support Chatbot Example Let’s use a customer support chatbot to understand this better. Consider a user-support chatbot which can help customers with: - Order tracking - Returns and refunds - Product information - Delivery issues Now, consider the following conversation: User: I want to return my keyboard. Chatbot: Sure. Please provide the order number. User: ORD-772. The application can now call a tool such as: lookup order "ORD-772" The tool can get the order details from an external database and may return a large JSON response: { "order id": "ORD-772", "customer id": "CUS-123", "product": "Mechanical Keyboard", "price": 89.99, "status": "Delivered", "delivery date": "2026-09-20", "payment method": "Credit Card", "warehouse": "...", "shipping details": "...", "internal metadata": "..." } As the conversation grows, the application can also accumulate a large amount of conversation history, retrieved documents, and tool outputs. If all of this information is passed to the model, it can lead to: - Too much noise - Conflicting information - Exceeding the context window - Stale information - Increasing token usage So we need a way to decide what should actually go into the context. What Goes Into the Context? We can divide the information available to the model into several layers. For example: - System instructions - Customer information - Current query - Conversation history - Retrieved information if any - Tool outputs Each of these layers has a different purpose. Some information is relatively stable and predictable, while other information can change significantly between user queries. Context Assembler Once the required information has been selected and transformed, it needs to be assembled into a format that the model can consume. The context assembler takes the selected information from the different sources and creates the final model input. For example: SYSTEM INSTRUCTIONS You are a customer support assistant. Be helpful and concise. Do not make up information. CUSTOMER DETAILS Customer tier: Gold Authenticated: Yes CURRENT QUERY Can I return this keyboard? RELEVANT POLICY DOCUMENTS RETRIEVED Customers can return eligible products within 30 days... CURRENT ORDER Order: ORD-772 Product: Mechanical Keyboard Status: Delivered Return eligible: Yes RECENT CONVERSATION User: I want to return my keyboard. Assistant: Sure, please provide your order number. User: ORD-772. But there is still one more constraint. Even after selecting the relevant information, we cannot keep adding information indefinitely. We have a limited token budget. How Do We Manage the Context? Now we need to answer another question: How much information can actually be included in the final context? This is where token counting and budgeting become important. Token Counter Before sending the request, we need to know how large the context is. The token counter measures the tokens used by each context layer. For example: - System instructions → 400 tokens - Customer details → 200 tokens - Conversation history → 1,500 tokens - Retrieved documents → 1,000 tokens - Tool outputs → 500 tokens - Current query → 200 tokens Total: 3,800 tokens The token counter helps the application understand how much of the available context budget is currently being used. But it does not decide what should be removed. That is the responsibility of the budget manager. Baseline Context vs Dynamic Context Before undestanding Budget Manager, lets understand different types of context. Not every layer should be treated in the same way. Some information is relatively fixed throughout the application. For example, the system instructions can remain almost the same across many requests. Similarly, customer information can often be narrowed down to only the fields required by the application For example, only consider username, user email, user tier etc . The tokens consumed by these layers will be relatively predictable, although there can still be some variation depending on the user or application state. These can be considered baseline context . Other information is much more dynamic and can change between every user query. For example: - Conversation histories - Retrieved policies - Tool observations - Current query These data can change significantly after every request. These can be considered dynamic context . Suppose the system instructions require 500 tokens. That is a relatively predictable cost. But a retriever may return 10 relevant chunks for one query and only 2 for another. The amount of context required therefore changes from query to query. The budget should therefore be flexible. Budget Manager Now the budget manager decides how much information can actually be included in the final context. Suppose we have allocated an input budget of 6,000 tokens. The budget manager needs to determine how much of each layer can fit within this limit. For example: - System instructions → 500 tokens - Customer information → 300 tokens - Current query → 200 tokens - Conversation history → 1,500 tokens - Retrieved documents → 2,000 tokens - Tool observations → 1,000 tokens - Buffer → 500 tokens Total: 6,000 tokens The important point is that the budget manager should not simply divide the available tokens equally across all layers. Different layers have different levels of importance and flexibility. For example: - System instructions may have a minimum required size and cannot be reduced much further. - Customer information may require only a few important fields. - Conversation history can be compressed or reduced to the most relevant information. - Retrieved documents can be reduced by selecting fewer or most relevant chunks. - Tool outputs can be transformed into a smaller structured representation. This gives the application some flexibility when the requirements of the current query change. Suppose the current query requires a large number of retrieved documents. The budget manager may allow the retrieval layer to use more tokens. But it cannot keep increasing the retrieval budget indefinitely. It may need to reduce the conversation history or tool observations to make room for the additional information. Similarly, if a tool returns a very large response, the tool output should first be reduced to the information required for the current task. If it is still too large, the budget manager can reduce the amount of tool information included in the final context. Overall Architecture and Flow Putting everything together, the overall flow can look like this: - User inputs a question - The user query determines what information may be required. - The application gathers information from different context sources. - The token counter measures how much token is currently being used by each source - The budget manager determines how much information can fit within the available budget. - Finally, the Context Assembler organizes the selected information into the final model input. - This final context is then sent to the LLM. Conclusion Context engineering is not limited to a single architecture but the whole idea is to ensure that the model receives the most relevant information within the available context window. Depending on the use case different methods like context compression, summarization, reranking etc can be used for context engineering. The workflow discussed in this article is one way to approach context management, illustrating how different components can work together to build the context required for a given task. Thank you for reading this post Let me know if you liked it, have questions, or spotted an error. Please feel free to contact or follow me through LinkedIn https://www.linkedin.com/in/vinitha-v-n-5a0560179/ .