The reality is that an agent doesn't need a verbatim transcript of its failures; it needs the current state of the investigation.
The Context Bottleneck #
When an agent fills its window with useless intermediate steps, we see a predictable slide in performance: slower processing times, higher latency, and the "lost in the middle" phenomenon where the LLM forgets the original goal because it's buried under 10k tokens of grep
results.
If we can compress that 40,000-token history into a concise status report, the agent stays sharp. Instead of a wall of text, the prompt should look like this:
Goal: Fix the API 500 error.
Found: Error started after deployment v1.4; DB is healthy; PAYMENT_API_KEY is missing.
Tried: Restarting the service (no effect).
Next: Fix environment config and retest.
Practical Strategies for Context Compression #
Depending on the complexity of the AI workflow, I've found three ways to handle this.
1. Pruning
This is the most aggressive approach. You simply delete the "dead ends." If an agent searched for
config.yaml
and found nothing, then searched settings.yaml
and found nothing, those failed attempts are baggage. Once the agent identifies the actual culprit—say, a missing environment variable—the previous failed searches provide zero value. Pruning removes the noise while keeping the "aha!" moment.
2. Distillation
For longer-running tasks, I prefer converting the raw conversation into a structured state. Instead of a chat history, the agent maintains a "state document." I've found that using a specific schema helps the LLM maintain coherence:
{
"current_goal": "Fix API 500 error",
"verified_facts": [
"Database connectivity is stable",
"Regression started at v1.4",
"PAYMENT_API_KEY is null"
],
"discarded_hypotheses": [
"Database timeout",
"Network partition"
],
"pending_actions": [
"Inject missing API key",
"Verify endpoint response"
]
}
By updating this JSON block after every few turns and wiping the intermediate tool outputs, you can keep the context window lean regardless of how many hours the agent has been working.
3. Generalisation
This is the "long-term memory" play. If an agent spends two hours figuring out that a specific legacy module requires a weird flag to run, that's not just a "fact" for this ticket—it's reusable knowledge.
Generalisation involves extracting a rule from a specific experience:
Specific:"Missing API key caused the payment API to fail in the staging env."** General:**"Always verify environment variable injection during v1.4+ deployments."
Implementation Detail: The Compression Trigger #
One technical hurdle is deciding when to compress. If you do it every turn, you waste tokens on the compression prompt itself. In our internal deployment, we use a token-threshold trigger.
CONTEXT_THRESHOLD = 15000
def manage_context(history):
current_tokens = count_tokens(history)
if current_tokens > CONTEXT_THRESHOLD:
summary = distill_history(history)
return [summary]
return history
This ensures we only pay the "compression tax" when the performance hit of a bloated context outweighs the cost of the distillation call. It's a practical way to build a production-ready LLM agent that doesn't lose the plot halfway through a complex bug hunt.
Next AI Agent Budgeting: Why I Built a Circuit Breaker →