All models, including large models such as Claude Sonnet, can have a limited context window. The context window is nothing but everything the model can see in one call.
Larger models can have a relatively bigger context window compared to smaller models. A bigger context size can hold more history compared to a smaller one.
We are going to see how we can manage the context size, regardless of the model's context size.
Generally, all conversation history will be stored in the local machine's RAM. As RAM size is limited, we cannot store all the history. So, we can calculate the tokens that are going to be stored in the RAM approximately using a package called count_tokens_approximately.
messages : 4
tokens used : 87
room left : 6913
turn 5: this call = 287 tokens, all calls so far = 1035 tokens
turn 20: this call = 887 tokens, all calls so far = 10140 tokens
turn 50: this call = 2087 tokens, all calls so far = 55350 tokens
So, we need to plan the budget for the context ahead. Instead of reacting when the window is full, decide in advance how much room each part gets:
WINDOW = 8_000
BUDGET = {"system": 0.05, "memories": 0.10, "retrieved": 0.35, "history": 0.40, "reserve": 0.10}
Here, we can define that the system can occupy 5% of the total usage. Likewise, memories contain important facts/information about the particular user, retrieved chunks are retrieved from RAG, and recent history/conversation is stored as part of the context.
Set an alarm when we reach the allotted memory usage so we can take action before overflow. What type of action can we take? We can trim the history, clear the old tool outputs, and perform summarization for the conversation.
Trimming - drops whole messages until the history fits. It can trim either at the end, beginning, or middle.
Clear the old tool outputs - keep the newest tool output in full and replace older ones with a one-line placeholder. Keep the same tool_call_id, so the AI message that asked for the tool still has its answer.
Summarization - trimming forgets. A rolling summary compresses the old part into a short note, so facts survive. For summarization, we will use a low-cost LLM. The prompt used for summarization tells what to keep: names, numbers, decisions, and preferences.
Choose the actions from the cheapest fix first, which means first try to clear the old tool outputs. If that is not sufficient, then move to trimming and then summarization, and store the whole history in long-term memory.