Managing Small Context Windows in Language Models

Managing small context windows in large language models can reduce API costs, lower latency, and avoid the 'lost in the middle' problem, according to a technical article that presents three strategies: sliding window truncation, token budgeting with retrieval-augmented generation, and specialized methods like rolling summaries and prompt compression. The article includes Python code demonstrating a sliding window memory class that keeps token usage flat by dropping the oldest conversation turns beyond a set maximum.

In this article, you will learn three practical strategies for managing small context windows in large language models, along with working Python examples that demonstrate how two of those strategies are implemented. Topics we will cover include: - How context truncation via the sliding window approach keeps token usage flat and predictable. - How token budgeting combined with retrieval-augmented generation ensures only the most relevant context fits within a prompt. - A concise overview of additional strategies for more specialized use cases — rolling summaries, prompt compression, and observation masking. Introduction Top-tier AI industries have become somewhat obsessed with language models capable of ingesting massive context windows , e.g. an entire book in a single prompt. However, what they won’t admit easily is that in real-world LLM applications, these massive context windows come with various limitations and challenges, including soaring API costs, unacceptable response times, and even worse, the so-called “lost in the middle” problem whereby a model ignores data deeply buried in the middle of the giant prompt. No surprise, then, that working with small yet smartly managed context windows could yield superior outcomes, reducing latency, minimizing costs, and forcing the model to concentrate on what truly matters to generate its response. This article unveils three of the most widely adopted practical strategies for managing and mastering small context windows in language models, along with examples that mimic the implementation of some of them for better understanding. Context Truncation: Sliding Window There is a consensus that sliding windows are arguably the most common and simplest strategy for managing shortened context windows in language models. Instead of providing an entire user conversation history to the model, the context is treated as a FIFO First-In-First-Out queue: as new interactions exchanged messages come in, the oldest ones are simply dropped. All it takes is defining the size of the context window and striking a balance between sufficient past context retention and latency-cost control. The main advantage of truncating the context via sliding windows is absolute control and predictability over token usage and computing overhead. The maximum number of interactions dealt with by the model at a given time remains fixed, keeping latency flat and surprise-free. To better understand how this approach works, let’s look at the following Python code in which you can freely adjust the value of max turns context window size and see how it affects the “memory” injected into the current prompt: python class SlidingWindowMemory: def init self, max turns=3 : """Keep only the last max turns of a conversation.""" self.max turns = max turns self.history = def add interaction self, user text, ai text : self.history.append {"user": user text, "ai": ai text} The logic behind a sliding window: drop the oldest turns if limits are surpassed if len self.history self.max turns: self.history = self.history -self.max turns: def build prompt self, new query : prompt = "System: Answer concisely based on recent context.\n\n" for turn in self.history: prompt += f"User: {turn 'user' }\nAI: {turn 'ai' }\n" prompt += f"User: {new query}\nAI:" return prompt --- Testing the Sliding Window mechanism: feel free to adjust the value of max turns --- memory = SlidingWindowMemory max turns=2 Simulating a long conversation memory.add interaction "Hi, I'm learning Python.", "Great choice " memory.add interaction "What are lists?", "Lists are mutable arrays." memory.add interaction "Can they hold mixed types?", "Yes, they can." The prompt will only contain the last 'max turns' interactions, saving tokens print memory.build prompt "How do I append to one?" 123456789101112131415161718192021222324252627282930 class SlidingWindowMemory: def init self, max turns=3 : """Keep only the last max turns of a conversation.""" self.max turns = max turns self.history = def add interaction self, user text, ai text : self.history.append {"user": user text, "ai": ai text} The logic behind a sliding window: drop the oldest turns if limits are surpassed if len self.history self.max turns: self.history = self.history -self.max turns: def build prompt self, new query : prompt = "System: Answer concisely based on recent context.\n\n" for turn in self.history: prompt += f"User: {turn 'user' }\nAI: {turn 'ai' }\n" prompt += f"User: {new query}\nAI:" return prompt --- Testing the Sliding Window mechanism: feel free to adjust the value of max turns ---memory = SlidingWindowMemory max turns=2 Simulating a long conversationmemory.add interaction "Hi, I'm learning Python.", "Great choice " memory.add interaction "What are lists?", "Lists are mutable arrays." memory.add interaction "Can they hold mixed types?", "Yes, they can." The prompt will only contain the last 'max turns' interactions, saving tokensprint memory.build prompt "How do I append to one?" Output: System: Answer concisely based on recent context. User: What are lists? AI: Lists are mutable arrays. User: Can they hold mixed types? AI: Yes, they can. User: How do I append to one? AI: 12345678 System: Answer concisely based on recent context. User: What are lists?AI: Lists are mutable arrays.User: Can they hold mixed types?AI: Yes, they can.User: How do I append to one?AI: You can also try extending the conversation history by appending new memory.add interaction calls with extra query-response pairs of your own, to test the mechanism for larger context windows. Token Budgeting and RAG Retrieval-Augmented Generation RAG systems supplement LLMs with engines that reference and retrieve external documents to enrich the original user prompt with founded, relevant context. Small context windows may intuitively force a ruthless attitude toward the data to include in the context. To address this, token budgeting splits the context window into zones with strict limits per zone. For instance, a token budgeting criterion could allow up to 20% of the context for system instructions, 20% for the chat history including the latest user query , and the remaining 60% for retrieved data. This incorporates a more dynamic retrieval and data chunking behavior, halting insertion as soon as budget limits are hit. The main advantage of token budgeting is preventing unduly large retrieved documents from quickly exhausting the prompt and ensuring only highly relevant, concentrated information is included, thus avoiding side issues like the aforementioned “lost in the middle” problem. This code excerpt exemplifies the use of the mechanism in Python, using a simple word count as a free, lightweight proxy for token budgeting — to make it more realistic, you could consider the commonly accepted heuristic of 1 word = 1.3 tokens on average. The loop inside the function shows how to reliably pack a prompt without surpassing enforced limits: python def build budgeted prompt system prompt, retrieved chunks, user query, max words=50 : """Packs context chunks into a prompt until a strict word budget is hit.""" Calculating the fixed cost of mandatory elements base words = len system prompt.split + len user query.split current words = base words included chunks = for chunk in retrieved chunks: chunk words = len chunk.split Only add the chunk if it fits within the strict budget if current words + chunk words <= max words: included chunks.append chunk current words += chunk words else: print f"Budget hit Left out {len retrieved chunks - len included chunks } chunks." break context str = "\n---\n".join included chunks return f"{system prompt}\n\nContext:\n{context str}\n\nUser: {user query}" --- Testing the Budgeted Prompt Mechanism --- system msg = "Use the context to answer." query = "What is the capital of Spain?" docs = "Seville is a city in Andalusia, Spain.", "Madrid is the capital of Spain.", We want this to fit "Spain is located in Southwestern Europe.", This might get cut off "The population of Spain is roughly 47 million." Setting a very small budget to see the cutoff in action print build budgeted prompt system msg, docs, query, max words=30 12345678910111213141516171819202122232425262728293031323334 def build budgeted prompt system prompt, retrieved chunks, user query, max words=50 : """Packs context chunks into a prompt until a strict word budget is hit.""" Calculating the fixed cost of mandatory elements base words = len system prompt.split + len user query.split current words = base words included chunks = for chunk in retrieved chunks: chunk words = len chunk.split Only add the chunk if it fits within the strict budget if current words + chunk words <= max words: included chunks.append chunk current words += chunk words else: print f"Budget hit Left out {len retrieved chunks - len included chunks } chunks." break context str = "\n---\n".join included chunks return f"{system prompt}\n\nContext:\n{context str}\n\nUser: {user query}" --- Testing the Budgeted Prompt Mechanism ---system msg = "Use the context to answer."query = "What is the capital of Spain?"docs = "Seville is a city in Andalusia, Spain.", "Madrid is the capital of Spain.", We want this to fit "Spain is located in Southwestern Europe.", This might get cut off "The population of Spain is roughly 47 million." Setting a very small budget to see the cutoff in actionprint build budgeted prompt system msg, docs, query, max words=30 Output: Budget hit Left out 1 chunks. Use the context to answer. Context: Seville is a city in Andalusia, Spain. --- Madrid is the capital of Spain. --- Spain is located in Southwestern Europe. User: What is the capital of Spain? 1234567891011 Budget hit Left out 1 chunks.Use the context to answer. Context:Seville is a city in Andalusia, Spain.---Madrid is the capital of Spain.---Spain is located in Southwestern Europe. User: What is the capital of Spain? Beyond the Basics: Other Strategies To close out, let’s quickly outline some other strategies for managing small context windows, particularly for specialized use cases. Be aware that some of these strategies typically require live API calls or additional external dependencies for their implementation. Rolling Summaries: This method uses an auxiliary LLM for summarization that condenses older conversation history into a compact paragraph, replacing the raw prompt text. It helps retain long-term memory without token bloat, but requires extra API calls to request and obtain the summaries, introducing added overhead and potential costs. Prompt Compression: Instead of resorting to an auxiliary model, an algorithm is invoked to strip out filler words, redundant data, and stop words from the raw context before feeding it to the main model. This can drastically reduce latency without compromising input quality or semantic intent, but if applied too aggressively, it could strip away subtle yet valuable nuances needed by the model to generate an acceptable response. Observation Masking: This approach evaluates the context to hide or mask older, structural noise — such as database queries in agent-based systems or intermediate code execution logs — while the core logic remains intact. It is a popular technique in autonomous agents fueled by LLMs, allowing them to stay focused on their immediate goal without being distracted by past internal steps. However, it is more complex to implement, as it requires determining which observations are safe to mask without compromising the agent’s reasoning chain. Closing Remarks Small context windows shouldn’t be regarded as a limitation but rather as an architectural feature for preventing major issues like excessive cost and latency. This article presented a number of strategies for effectively managing small context windows in LLMs to yield faster and cheaper solutions without compromising accuracy.