Posted August 30, 2026
In part one, I told you the boring technique won. I had built a clever cache-preserving prefix fork for AI context compaction, benchmarked it honestly, and watched it lose to the unglamorous production baseline: a fresh compactor that searches canonical history, retrieves evidence, and publishes a validated handoff. I ended that post saying the boring decision was the one I trusted.
Sixty days of production data later, I have to report something uncomfortable.
The boring winner broke too.
This is the part two where I show you exactly how it broke, what the postmortem taxonomy looked like, how I ran an eval-driven redesign against real production failures, and why the replacement is somehow even more boring than the thing it replaces.
There is a ladder of boring in this field. I keep climbing down it, and the system keeps getting better.
What sixty days of production said #
Friday runs compaction whenever a chat's context grows past an economic threshold. The mechanism that won part one's benchmark — I called it search + memory — became the production default: a compactor fork gets a search tool over canonical history plus publish/edit/read tools for its draft, up to 50 tool turns, a strict evidence-claim schema with citations to exact message ids, and an acceptance gate requiring the projected request to land at or below a headroom target.
Over sixty days, the accounting came back roughly like this: about 136 accepted compactions, about 58 terminal failures, and about 132 superseded attempts. Call it a ~30% terminal failure rate on the attempts that mattered.
Worse than the rate was the shape of the cost curve. Failures cost more than successes. On one provider, failed attempts averaged 6.36 million input tokens, with a maximum of 11.7 million. On another, failures averaged 18.5 minutes of wall time, maxing out at 77 minutes. The system spent its most extravagant effort producing nothing.
When I pulled the failed attempts and read them, they sorted into five classes:
| Class | What actually happened |
|---|---|
| Retrieval spirals | One attempt spent all 50 tool turns calling search, never published once, and burned 6.94M input tokens in nine minutes. Others spent 35–39 turns re-reading their own draft. |
| Structural floors | Messages pinned raw by policy (unresolved errors, unsafe tool state) exceeded the headroom target by themselves — before the summary document contributed a single token. One chat failed identically four times in one night because this floor failure was marked retryable. |
| Too late | Chats grew past the compactor's own request limit. No loop, however clever, can fix this; the compactor cannot even read its input. |
| Provider churn | Rate limits and outages retried without meaningful backoff — four failures in ninety seconds. |
| Emission traps | The document a model needed to write was bigger than its output cap, so it truncated, so the whole candidate was discarded. Mid-tier models also just failed the claim schema: "Evidence claim is not an object." |
Here is the diagnosis I eventually wrote down, and it is the sentence that drove everything after: an emergency agentic loop with strict schema-and-target acceptance, run at the worst possible moment, by whatever model the chat happens to be using — where every guard we added extended the loop instead of ending it.
The technique didn't fail. The regime failed. Search + memory won part one's benchmark, which measured handoff quality under controlled conditions. Production measured something else: what happens when you ask for that handoff at the cliff edge, from a stressed context, with a validator that treats "good but slightly too big" the same as "garbage."
The redesign day #
I was on a road trip, so I dispatched the redesign to an agent with a mission brief and hard constraints: production database read-only, private transcripts never leave gitignored directories, no runtime changes shipped unattended, an explicit token budget for eval spend. It ran the whole program autonomously and kept an append-only log of every assumption. What follows is what it did and what the numbers said.
First: real fixtures. Eight production chats, exported with their full message histories and compaction-attempt rows — one exemplar for each failure class, plus a success-path baseline and the biggest recent chat (17MB of transcript). Not synthetic. The existing live test for this system used a five-message fixture totaling about 2,000 tokens; production was failing at 460,000+. That gap between what the eval exercised and what production experienced is where the whole problem hid.
The export immediately produced the most important number of the day. Byte composition by content type, across all eight chats: tool calls and tool payloads were 70–99% of the bytes. Human-and-model conversation text was about 1%.
Sit with that. The thing we were asking a model to carefully summarize was two orders of magnitude more tool exhaust than conversation.
Second: a survey. How do the other shipping harnesses do this? Six codebases read end to end — opencode, OpenAI's codex CLI, gemini-cli, goose, aider, cline — plus the published behavior of Claude Code, Anthropic's context-management API, and MemGPT/Letta.
The convergence was embarrassing, in the way good surveys are:
Nobody uses an agentic loop. The most expensive mechanism found anywhere was two model calls (gemini-cli does a summary plus one self-critique pass). Everyone else: one call, or zero.Everybody triggers early. gemini-cli compacts at 50% of the context window. goose at 80%. codex and cline at 90%. Nobody waits for the emergency.Everybody decays old tool results deterministically first. Placeholder substitution, spill-to-disk, middle-out dropping — every serious harness clears tool exhaust with plain code before any model gets involved. Claude Code's microcompaction, Anthropic'sclear_tool_uses
API, gemini-cli's masking service, codex's output blanking: this is the industry's rung one.Acceptance is lenient everywhere. The strictest gate found: non-empty, smaller than before, under the recovery target — and on failure, cline falls back to azero-LLM deterministiccompactor. Nobody rejects a candidate for being over an internal headroom goal while still being dramatically smaller than the request it replaces. Nobody demands per-claim JSON citations.
My strict claim schema, 50-turn loop, and hard headroom gate existed nowhere else in the field.
Third: candidates, run against the real fixtures. A production-parity eval harness — same message shapes, same provider adapters, same budget math as the runtime — with the compaction mechanism swapped per arm:
M0: the current production loop, as-is (the baseline).** M1**: one single-shot call producing a structured markdown snapshot. No tools. No claim JSON. One size-feedback retry allowed.M2: deterministic elision first — old tool calls reduced to name + truncated args + a short result digest, thinking dropped, media stubbed, recent window untouched — then M1 on the remainder.M3: a rolling replay of the whole transcript, folding the oldest turns into a carried-forward snapshot every time projected size crossed ~150k tokens. This simulatesnever having an emergency at all.M4: not a new mechanism — a policy overlay asking how many failures become successes if acceptance is just "smaller and sendable."
What the matrix said #
Deterministic elision alone recovered 69–95% of tokens on every fixture, with zero model calls. Six of eight landed under the economic acceptance ceiling before any model was involved. One landed under the strict headroom target — and it was the fixture from the "too late" class, the chat that had grown past the compactor's own request limit, the one where production compaction was mathematically impossible. Plain code solved it in milliseconds.
Given the 70–99% tool-exhaust composition, this shouldn't have been surprising. It still felt like a magic trick.
Single-shot at emergency time worked when it could read the source, and honestly reported when it couldn't. On the three fixtures whose raw history still fit in the compactor's request limit, M1 accepted with one call, ~2.5 minutes, and huge reductions. On the other five, it failed instantly at preflight — the raw source was bigger than the model's request limit. Which is the correct behavior, and also the point: at emergency time, a single-shot summarizer often cannot exist without deterministic decay in front of it.
Elision plus single-shot accepted everything. Eight for eight, under the strict target, at most one provider call each, 60–163 seconds:
| Method | Accepted | Median wall | Median input tokens | Provider calls |
|---|---|---|---|---|
| M0 current loop (2 fixtures) | 2/2 | 460s | 671k | 13–17 |
| M1 single-shot at emergency | 3/8 | 146s | 404k | 1 |
| M2 elision → single-shot | 8/8 | |||
| 81s | ||||
| 165k | ||||
| ≤1 | ||||
| M3 rolling folds (3 fixtures) | 3/3 | per-fold ~40–300s | 40k–417k lifetime | |
| 2–15 lifetime |
The rolling simulation made the emergency unrepresentable. Replaying the "too late" chat with incremental folding, its projected context never exceeded 185,644 tokens across its entire life. The state that killed production compaction never comes into existence. And the total token cost of all compaction across that chat's lifetime was less than one percent of what a single production failure had burned.
The rolling arm also caught its own gap, which I appreciated: one fixture had a single turn containing multi-megabyte script outputs, and folding at turn boundaries pushed one fold's source past the request limit. The fix was already on the board — run elision inside the fold step too. With that composed, three for three, and fold costs dropped another 7x.
And the baseline? I ran the current production loop on two fixtures for a head-to-head. It succeeded on both — thirteen and seventeen provider calls, 428 and 492 seconds, roughly 670,000 input tokens each, versus one call, 62–163 seconds, and 85–165k tokens for the composed candidate on the same chats. Its one genuine edge: the loop's citation discipline retained somewhat more exact identifiers in the document (a probe-based fidelity check gave it 0.47 versus 0.40 on one fixture). But the elision digests retain 0.67–1.0 of those identifiers for free, sitting right next to the document in the projection. The loop's fidelity edge is real and also already paid for by rung one.
A judge model scored continuation safety 5/5 on nearly every accepted candidate. The one score of 1 was a catch, not an embarrassment: the judge noticed the projection boundary had severed an in-flight tool call from its result. That's now a hard constraint in the redesign — never cut between a tool call and its result — which cline, it turns out, already learned.
The new shape #
Four rungs, replacing the loop:
Deterministic decay, always on. Old tool traffic beyond a protected recent window becomes name + args + status + a short digest. Plain code. Cannot fail. On our byte mix it does 69–95% of the work, and itisthe permanent fallback.One structured snapshot call. Markdown sections — goal, decisions and constraints, tool evidence, open errors, completed work, next actions, retrieval hints — with prior snapshots merged forward. No tools. No claim schema. A truncated document is a slightly worse document, not a discarded candidate.Lenient acceptance. Smaller and sendable is accepted; the headroom target is what the size-retry aims for, not a gate that converts degradation into outage. If the model call fails, ship the elision-only projection. Terminal failure is reserved for "even plain code can't make this sendable," which no fixture exhibited.Early, incremental scheduling. Fold at ~150k projected tokens, not at the cliff. The emergency regime stops being a separate, more dangerous mode of operation because it stops being reachable.
Each failure class from the postmortem maps to a rung that kills it. Spirals die because there is no loop. Floors die because open errors live in a document section instead of raw-pinned messages. "Too late" dies because early folding makes the state unreachable — and even at the cliff, elision alone rescued it. Provider churn stops mattering because the fallback needs no provider. Emission traps die because there is no schema to fail.
The ladder of boring #
Part one's lesson was that a clever idea has to beat the boring incumbent on a fair benchmark, and mine didn't.
Part two's lesson is sharper: the benchmark that crowned the incumbent was itself too kind, because it never measured the regime — the moment of invocation, the acceptance policy, the failure economics. Search + memory produced better handoffs than my clever fork. It also produced its handoffs through fifty-turn tool loops with a strict validator at the worst possible moment, and under production's real distribution of models and chat shapes, that machinery failed expensively about a third of the time, and its failures cost more than its successes.
The replacement is the most boring artifact I have ever endorsed: delete old tool output, ask for one summary, accept it if it's smaller, do it before you're desperate. Every shipping harness I surveyed had already converged there. I got to the same place the long way — through a clever fork that lost, a validated loop that won and then broke, and an eval matrix over real corpses from my own production database.
Updates to the checklist from part one, earned the hard way:
Evaluate the regime, not just the technique. When it runs, what accepts it, and what failure costs are part of the design.Composition is destiny. Measure what your context is actually made of before choosing what intelligence to apply to it. Ours was 99% tool exhaust; the answer wassubstring
, not a smarter agent.A strict gate on a lossy process converts degradation into outage. Smaller-but-over-target is a success wearing a failure's clothes.Your fallback should be incapable of failing. Deterministic elision changed the failure math more than any model improvement could.A guard that extends the loop is fuel. Guards should end things.
The runtime change ships next, through the ordinary review gates, with the eval harness standing guard. The loop that won part one goes away entirely — no flags, no parallel paths. It was a good baseline. It lost to something more boring, which by now I have learned to read as a compliment.
Same model economics as last time, for the curious: the whole redesign matrix — every arm, every fixture, judge included — cost about 9.5 million input tokens on Qwen 3.8 Max through Alibaba's Token Plan, which is to say: two production failures' worth.
Sources and further reading #
Part one: I Tested AI Context Compaction. The Clever Fork Lost.Anthropic: Effective context engineering for AI agentsAnthropic context editing / tool-result clearing APIClaude Code: context windows and compactionMemGPT: Towards LLMs as Operating SystemsFactory.ai: Evaluating compressionLangChain: Context management for Deep AgentsManus: Context engineering lessons- Surveyed harnesses: opencode,codex,gemini-cli,goose,aider,cline