{"slug": "how-to-set-an-ai-agent-context-window-budget-before-long-tasks-fall-apart", "title": "How to set an AI agent context window budget before long tasks fall apart", "summary": "Chroma's research team published a July 2025 technical report, Context Rot, testing 18 models including GPT-4.1, Claude 4 and Gemini 2.5, and found reliability fell as input length grew even on simple tasks. Anthropic's engineering team reported in a September 2025 post that clearing old tool results cut token use by 84% in a 100-turn web search test, and the article recommends budgeting a 200,000-token window with a 40,000-token reserve, pruning at 100,000 tokens and a handoff summary at 130,000. The guidance matters because agents re-send full history every turn, so stale tool results inflate both error rates and inference cost as tasks run long.", "body_md": "*A bigger window won't save an agent that's drowning in its own old search results. The fix that works sounds backwards: cap how much it's allowed to remember.*\n\n- Agent quality drops as the window fills with stale tool results, so a larger window doesn't fix a long task.\n- Chroma's 2025 Context Rot report tested 18 models and found reliability fell as input length grew, even on simple tasks.\n- On a 200,000-token window, a workable starting budget is a 40,000-token reserve, pruning at 100,000 and a handoff summary at 130,000.\n- Anthropic reported that clearing old tool results cut token use by 84% in a 100-turn web search test.\n- Prune in batches rather than every turn, because rewriting history breaks the prompt cache that makes repeated input cheap.\n\nYour ai agent context window budget probably doesn't exist, and that's why the agent that handled step three flawlessly is fumbling step forty. Nothing broke. It's the same model with the same prompt. What changed is what it's reading: by step forty the window holds thirty-odd tool results, most of them stale, and every new decision gets made on top of them.\n\nChroma's research team published a technical report in July 2025 called Context Rot. It tested 18 models, including GPT-4.1, Claude 4 and Gemini 2.5, and found that performance became less reliable as input grew longer, even on simple tasks. The models didn't treat the 100,000th token the way they treated the hundredth. That fits an older finding from a Stanford-led paper, Lost in the Middle, which showed models use information at the start and end of a long input better than information in the middle.\n\nThink about where things land in an agent's window. The original goal sits at the top. The latest tool result sits at the bottom. The decision you made at step twelve, the one that explains why the agent is doing what it's doing at step forty, is buried in the middle with everything else.\n\nAnthropic's engineering team made the same point in its September 2025 post on context engineering. It describes context as a finite resource with diminishing returns, something closer to an attention budget than a storage bin. That's the right frame. Every token you add competes with every token already there.\n\n[How Context Window Length Actually Breaks Your AI Agent's Unit Economics](https://startupfortune.com/how-context-window-length-actually-breaks-your-ai-agents-unit-economics/)\n\nHow does context window length affect inference cost? Not linearly. Attention scales roughly with the square of context length, and the KV cache adds a growing memory tax on top, which is exactly where AI agent unit economics quietly fall apart. - [how context window size affects AI agent costs](https://startupfortune.com/how-context-window-length-actually-breaks-your-ai-agents-unit-economics/) - [why large language model context breaks unit economics](https://startupfortune.com/how-context-window-length-actually-breaks-your-ai-agents-unit-economics/)\n\nTool results do most of the damage. A web search can return 6,000 tokens. A file read can return 10,000. A failed test run with a full stack trace can return 8,000 more. The agent needed one line from each, and the rest stays in the window for the remainder of the task.\n\nWorse, old results go out of date. Say the agent reads a config file at step five, edits it at step nine, and reads it again at step fourteen. Now the window holds two versions of the same file, and nothing marks which one is current. The model has to work that out every single turn, and sometimes it guesses wrong.\n\nFailed attempts do their own harm. When the window is full of the agent's earlier wrong turns, it tends to repeat them, because they look like the pattern it's supposed to follow.\n\n## The bill grows faster than the task\n\nBloat costs money as well as accuracy. An agent doesn't remember anything between calls, so every turn re-sends the whole history. Suppose each step adds 4,000 tokens and the task runs 50 steps. The model reads 4,000 tokens on the first call, 8,000 on the second, and by the last call it's reading 200,000. Add those up and you've processed about 5.1 million input tokens for a task that only produced 200,000 of new material. At an illustrative $3 per million input tokens, that's roughly $15 for one run, before any caching.\n\nManus, the agent company, wrote in a 2025 post on lessons from building its product that the ratio of input tokens to output tokens in its agents averages around 100 to 1. If that sounds extreme, check your own logs. Most teams that look find the same shape.\n\n## A context window management for AI agents plan with three numbers\n\nHere's the method. It needs three thresholds, and I'll use a 200,000-token window as the example. These are starting points that I'd expect to work for tool-heavy agents. They aren't published standards, and you should test them on your own tasks, which I'll get to below.\n\nThe first number is the reserve. Take 20% of the window, 40,000 tokens here, and treat it as untouchable. It covers your system prompt, your tool definitions (a dozen tools with detailed schemas can run into the thousands of tokens by themselves), and room for the model's reasoning and its answer. Reasoning tokens count against the window too, so an agent that's allowed to think hard needs more headroom than one that isn't. Whatever's left is your working space, and you don't plan around the reserve at all.\n\n[Why Does My AI Coding Agent Bill Keep Increasing Overnight](https://startupfortune.com/why-does-my-ai-coding-agent-bill-keep-increasing-overnight/)\n\nWhy does my AI coding agent bill keep increasing when nobody touched the keyboard overnight? Usually it's a runaway tool-call loop compounding token costs with every retry. This guide walks through the mechanism, a real incident involving Replit's agent, and the specific guardrails that stop it. - [why AI coding agent bills spike overnight](https://startupfortune.com/why-does-my-ai-coding-agent-bill-keep-increasing-overnight/) - [preventing runaway tool call loops in agents](https://startupfortune.com/why-does-my-ai-coding-agent-bill-keep-increasing-overnight/)\n\nThe second number is the prune line. Set it at 50% of the window, which is 100,000 tokens. When the running total crosses it, the agent clears old tool results. Keep the last five or so intact, since those are the ones the model is probably still using. Replace everything older with a one-line stub: which tool was called, with what arguments, what was found, and where to fetch it again if needed. The record of what the agent did stays. The bulk of what it read goes.\n\nThe third number is the summarize line, at 65%, or 130,000 tokens. If pruning hasn't bought enough room, the agent stops and writes a handoff note, and a fresh context starts from that note plus the system prompt and the last couple of turns. Never plan to reach 80%. By then you're past the point where the model is doing its best work.\n\nWhy act at half the window rather than near the top? Because of when the summary gets written. Compaction near the limit means the model is summarizing its own work at the moment it's least reliable. A summary written at 130,000 tokens is a better document than one written at 195,000. Claude Code, for what it's worth, runs its own compaction as the window nears its limit, and anyone who's used it on a long session has seen the effect: the first compaction goes fine, and the third one starts dropping things.\n\n## What the pruning step should look like\n\nAnthropic shipped a version of this idea in September 2025 as context editing, which automatically clears stale tool calls and results as a conversation nears the limit. The company reported that in an internal test on a 100-turn web search task, context editing cut token consumption by 84% and let workflows finish that would otherwise have failed from running out of room. Paired with its memory tool, which lets the agent save notes outside the window, performance improved 39% over baseline. Editing alone improved it by 29%.\n\nThose are Anthropic's numbers from its own evaluation, so don't treat them as a promise for your agent. They do show which direction the result points: less context, better outcome.\n\nOne catch will bite you if you ignore it. Most providers cache the start of your prompt, and cached input is far cheaper than fresh input. Manus noted that cached tokens on Claude Sonnet cost $0.30 per million against $3 for uncached ones. Every time you rewrite the middle of your history, you break the cache from that point on. If you prune after every turn, you might save tokens and still pay more.\n\nSo prune in batches. Let the window grow until it crosses the line, clear a big chunk in one pass, and then run on the new, shorter prefix for many turns. Keep the system prompt and tool list byte-for-byte stable, and don't put a timestamp at the top of the prompt. It's a small detail that has cost real teams their cache hit rate.\n\n## The handoff note is where teams lose the thread\n\nDon't ask the model to summarize the conversation. That prompt gets you a smooth paragraph about what happened, and it'll drop the one detail the next stage needed. Use a fixed template instead and make every field mandatory: the goal in the user's own words, decisions made and the reason for each, files or records changed, approaches that failed and why, and the single next step.\n\nThe failed-approaches field matters most, and it's the one that gets skipped. Without it, the fresh context cheerfully tries the same dead end again.\n\nBetter still, don't wait for the summarize line to write anything down. Have the agent keep a progress file as it goes, a running notes document it updates after each meaningful step. Anthropic has described Claude playing Pokémon doing something similar, keeping its own tallies and notes across thousands of game steps, and picking up where it left off after a context reset. Notes outside the window cost almost nothing to keep and make every later reset cheap.\n\n## How to reduce context bloat in LLM agents before it starts\n\nPruning is cleanup. The cheaper move is to stop the junk from arriving. Look at what your tools return, because that's where most of the volume comes from. A search tool that returns five trimmed snippets instead of five full pages can save 20,000 tokens a call. A file reader that takes a line range beats one that dumps the whole file. A test runner should return the failures, not the 400 passing lines above them. If a tool returns HTML, strip it before the model sees it.\n\nSub-agents are the other lever. Anthropic described this in its June 2025 write-up of its multi-agent research system: a lead agent hands a chunk of work to a worker with its own clean window, and the worker comes back with a short summary instead of its whole trail. The lead never sees the 30 pages the worker read. It sees the 1,500 tokens the worker decided mattered.\n\nFrankly, that's the use for a big window. Giant contexts are for the job where you really do need one enormous document in view at once, like a long contract or a codebase audit. They aren't a reason to let history pile up unattended.\n\n## Test it before you trust it\n\nYou don't need to take my thresholds on faith, and you shouldn't. Take 20 tasks where you already know the right answer. Run them with the working ceiling set at 60,000 tokens, then 100,000, then 150,000, and score the results the same way each time. Track cost per completed task, not just cost per run, since a cheap run that fails costs more than an expensive one that works.\n\nMost teams find the curve flatter than they expected up to a point, and then it drops. That point is your real ceiling, and it's usually well short of the number printed on the model's spec sheet. Move your prune line to roughly half of it.\n\nLog the token count at every turn. If you do nothing else from this piece, do that, because a team that can see the count climbing will notice the first time an agent crosses 100,000 tokens on step 22 of a 25-step job. Pick your three numbers, write them into the agent's config, and check the logs after the first week of real runs.\n\n**Also read:** [How to calculate burn rate variance startup boards actually check](https://startupfortune.com/how-to-calculate-burn-rate-variance-startup-boards-actually-check/) • [How to Calculate Sales Cycle Length by Deal Size Before You Forecast](https://startupfortune.com/how-to-calculate-sales-cycle-length-by-deal-size-before-you-forecast/) • [How to Calculate Days Sales Outstanding for Startups Before Cash Runs Short](https://startupfortune.com/how-to-calculate-days-sales-outstanding-for-startups-before-cash-runs-short/)\n\n*This article is posted in [AI News](https://startupfortune.com/category/ai/), check it out for more related stories.*\n\n## Join the discussion\n\n[Open in the community →](https://startupfortune.com/community/)\n\nAlmost there. Sign in and your reply posts straight away.", "url": "https://wpnews.pro/news/how-to-set-an-ai-agent-context-window-budget-before-long-tasks-fall-apart", "canonical_source": "https://startupfortune.com/how-to-set-an-ai-agent-context-window-budget-before-long-tasks-fall-apart/", "published_at": "2026-11-15 03:06:46+00:00", "updated_at": "2026-11-15 03:20:51.253390+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-research", "ai-tools"], "entities": ["Chroma", "Anthropic", "GPT-4.1", "Claude 4", "Gemini 2.5", "Stanford"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-set-an-ai-agent-context-window-budget-before-long-tasks-fall-apart", "markdown": "https://wpnews.pro/news/how-to-set-an-ai-agent-context-window-budget-before-long-tasks-fall-apart.md", "text": "https://wpnews.pro/news/how-to-set-an-ai-agent-context-window-budget-before-long-tasks-fall-apart.txt", "jsonld": "https://wpnews.pro/news/how-to-set-an-ai-agent-context-window-budget-before-long-tasks-fall-apart.jsonld"}}