# I Tested AI Context Compaction. The Clever Fork Lost.

> Source: <https://rico.codes/ai-context-compaction-evals>
> Published: 2026-08-16 08:22:00+00:00

# I Tested AI Context Compaction. The Clever Fork Lost.

Posted August 16, 2026

I had a clever idea about AI context compaction.

It was actually clever.

It also lost.

That distinction matters because the agent world is full of ideas that feel so architecturally satisfying that people skip the annoying step where they prove the idea is worth shipping.

I did not want to do that.

The idea was this: when an AI conversation gets too large, do not start a fresh compaction request and paste the history into it. Fork the existing conversation at its exact prefix, append one new instruction asking the same model to write a context handoff, and let the provider reuse the cache it already built.

In theory, the fork gets two benefits at once:

- The model stays inside the same visible working context instead of reading a reconstructed transcript.
- The provider can reuse the expensive prefix instead of processing it again at full price.

This felt like the right shape. It preserves continuity. It respects how prompt caches work. It avoids shipping a giant inline transcript to a new worker.

So I built it.

Then I built an eval designed to make the current approach fail.

Then the current approach won.

## What compaction is actually trying to preserve

Long agent conversations eventually become economically stupid before they become technically impossible.

A model may support hundreds of thousands of tokens, but that does not mean every turn should keep paying to drag the entire history forward. Tool results accumulate. Old branches remain in the transcript. Completed work keeps consuming attention. Latency grows. Cache behavior gets more important. At some point the system needs a smaller handoff document that can carry the work into the next context window.

The lazy framing is that compaction is summarization.

It is not.

A useful compaction has to preserve the live state of the work:

- the active objective;
- decisions and why they were made;
- evidence supporting or weakening each hypothesis;
- constraints that still apply;
- constraints that were superseded;
- actions already completed;
- writes that must not be repeated;
- uncertainty that should remain uncertainty;
- the next discriminating action.

That is closer to a transaction log plus a project handoff than a summary.

If the original conversation says, “We tried A, evidence X disproved it, constraint Y changed, and the next move is B,” a bad compaction remembers A and Y as facts. A good compaction preserves the causal update: A is dead, Y is old, and B is next because of X.

This is also why I did not ask the benchmark to preserve hidden chain of thought. The model does not expose some pristine internal diary that can be copied into a new request. What the system can preserve and evaluate is provider-visible working state: concise reasoning notes, decisions, evidence, rejected paths, uncertainty, completed actions, and next steps.

That is the state that matters operationally anyway.

## The techniques

I compared several ways of producing the handoff.

| Technique | How the handoff gets made | What it is testing |
|---|---|---|
| Full history | Continue with the entire trajectory | Upper-bound control |
| Lossy summary | Compress aggressively without recovery | Lower-bound control |
| Search + memory | A fresh compactor searches bounded canonical history, retrieves evidence, then publishes a validated handoff | Current production baseline |
| Prefix-direct | Append a handoff instruction to an exact cached prefix and have that model publish directly | Continuity plus cache reuse |
| Prefix-plan-worker | The exact prefix fork writes a structured brief, then a fresh worker searches history and publishes the final handoff | Same-context planning plus controlled publication |
| Sandbox-tool fork | Let the prefix fork execute deterministic disposable tools | Whether tool access is the missing causal ingredient |

The phrase “exact cached prefix” is doing a lot of work here.

It is not enough for the logical Friday transcript to look the same. The provider-facing bytes have to remain the same too. System instructions, tool schemas, cache markers, message ordering, tool-call serialization, and repair turns can all change the wire representation.

I added adapter-level proofs for that. The active request had to be a byte-identical prefix of the compaction request. Repair turns had to extend the fork without rewriting the inherited source prefix. Even a compaction tool call could not be allowed to displace one of Qwen's explicit cache markers.

That work found real bugs. The first implementation preserved the application-level prompt hash while moving provider cache boundaries. It looked exact from inside the harness and was not exact on the wire.

This is why “we use prompt caching” is not evidence. Cache behavior is a provider contract, not a vibe.

## The hypothesis

My favored hypothesis was:

A byte-identical prefix fork will preserve more reasoning continuity than a fresh search-based compactor while staying within 1.5 times the baseline cost and latency because most of the inherited context will be cached.

I gave it real ship gates:

| Gate | Requirement |
|---|---|
| Continuity improvement | At least 15 points over current |
| Confidence | Paired 95% interval above zero |
| Pairwise judge preference | At least 60% of non-ties |
| Deterministic final-state success | No regression |
| Cache-adjusted cost | No more than 1.5x current |
| p95 compaction latency | No more than 1.5x current |
| Exact inherited prefix | 100% |
| Safety and duplicate writes | Zero failures |
| Prefix fallback | Zero |

That last gate became important later.

If a prefix experiment silently falls back to the existing search compactor, it can inherit the baseline's handoff quality while still being labeled a prefix success. That is benchmark contamination. The eval now records prefix fallbacks explicitly and makes any one of them a hard failure.

## I needed an eval that was not already solved

Needle-in-a-haystack tests are useful, but they were too easy for this question. A compaction can retain a literal fact and still destroy the work.

I borrowed the shape of the benchmark from several stronger eval traditions:

[LongMemEval-V2](https://xiaowu0162.github.io/longmemeval-v2/)for dynamic state, workflow knowledge, environment gotchas, and premise awareness.[NoLiMa](https://github.com/adobe-research/NoLiMa)for minimizing lexical overlap so retrieval requires latent association instead of keyword matching.[tau2-bench](https://github.com/sierra-research/tau2-bench/blob/main/docs/evaluation.md)for checking the exact final state rather than demanding one privileged path.[RULER](https://github.com/NVIDIA/RULER)and[LongBench v2](https://arxiv.org/abs/2412.15204)for configurable long-context complexity and reasoning rather than raw recall.[MT-Bench's judge study](https://arxiv.org/abs/2306.05685)for reference-guided judging and swapped answer order to expose position bias.

The resulting benchmark has five seeded families.

| Family | The reasoning state that must survive |
|---|---|
| Incident diagnosis | Evidence reverses an early hypothesis; choose the next discriminating test |
| Changing implementation plan | Preserve revised dependencies and do not resurrect a superseded design |
| Data reconciliation | Derive a source-precedence policy, absorb corrections, apply it to a paraphrased case |
| Interrupted tool workflow | Resume completed work, avoid duplicate writes, survive an environment gotcha |
| Product judgment | Preserve the user's rationale, taste, trust boundaries, and scope on a new proposal |

Each fixture carries a gold `ReasoningStateGraphV1`

. It names the active goal, hypotheses, evidence edges, active and superseded constraints, decisions, rationale, completed actions, unresolved uncertainty, forbidden repetitions, and expected next action.

The post-compaction challenge is not “what was the secret code from turn 40?”

It is closer to:

The deployment symptom changed in this specific way. Given the evidence already collected, what should you test next, what should you not repeat, and which earlier hypothesis is now superseded?

That is much harder to bluff.

## Deterministic scoring first, model judging second

I wanted most of the score to come from things a script could verify.

The composite was designed as:

```
continuity score = 60% deterministic outcome/state + 40% semantic judge
```

The deterministic side checks exact sandbox state, completed-action preservation, forbidden duplicate writes, active versus superseded constraints, and required next actions.

The semantic judge sees a bounded packet containing the gold state graph, the challenge, the handoff, and the continuation result. It does not see the arm name or cost. It must cite gold graph identifiers for every finding. Pairwise comparisons run in both answer orders; inconsistent preferences become ties.

Before the judge's scores count, it has to pass known corruptions: omitted rationale, resurrected constraints, false certainty, verbosity attacks, and identical-answer ties.

I do not think “LLM as a judge” is automatically scientific. I think a calibrated, blinded, bounded judge can cover semantic properties that are expensive to reduce to string matching, as long as deterministic state remains the authority where deterministic state exists.

## Calibration: prove the benchmark has room

Before comparing compaction techniques, I ran a 64K calibration.

| Control | Aggregate continuity | What it should prove |
|---|---|---|
| Full-history oracle | 97.32 | The tasks are solvable |
| Search + memory | 74.84 | The baseline is useful but not saturated |
| Deliberately lossy summary | 9.07 | The benchmark detects destroyed reasoning state |

No baseline family exceeded 84.4. The oracle's weakest family still scored 94.4. The judge calibration passed 6 out of 6 adversarial cases.

That was the green light.

The eval was not just measuring whether the model could do the task at all. It also was not an easy suite where every halfway competent compactor scored 98.

There was room for the clever idea to win.

## The cache part actually worked

Qwen's explicit context cache is real. Its [official context-cache documentation](https://docs.qwencloud.com/developer-guides/text-generation/context-cache) lets callers mark stable content with `cache_control`

. The published economics at the time of this experiment priced explicit writes at 125% of normal input and reads at 10%, with a short renewable lifetime.

In a direct repeated-prefix probe, the second request contained 5,438 input tokens and Qwen reported 5,432 cached tokens.

That is an excellent cache hit.

It is also not the whole economic story.

Qwen does not expose a free “save this prefix for later” endpoint. Creating the explicit cache requires a model request. The first pass pays the cache-creation premium. The compactor still has to generate and sometimes repair the handoff. If the attempt fails and falls back, the system pays for both paths.

Cache-adjusted cost is therefore closer to:

```
ordinary input
+ 1.25 * cache creation
+ 0.10 * cache reads
+ output and reasoning
+ retries
+ fallback
```

The cache hit can be nearly perfect while the product route is still expensive.

That is exactly what happened.

## The result

The cleanest exact-save comparison looked like this:

| Measure | Search + memory | Exact prefix fork | Fork vs. baseline |
|---|---|---|---|
| Deterministic continuity | 60 |
40 | -20 points |
| Input tokens | 18,254 | 224,686 | 12.3x |
| Cached input tokens | 3,695 | 133,431 | 36.1x |
| Cache creation tokens | 14,541 | 91,219 | 6.3x |
| Cache-adjusted input cost | 18,563.75 |
127,402.85 | 6.9x |
| Compaction latency | 75.7s |
682.3s | 9.0x |
| Exact inherited prefix | Not applicable | 2 of 2 | Passed |
| Prefix fallback | 0 | 1 | Hard failure |

The fork did the technically impressive thing. It preserved the exact inherited prefix. It produced large cache reads. It proved the provider adapter could keep its markers stable across an appended compaction request and a repair turn.

Then it failed to publish a valid handoff in its two allowed attempts and fell back to search.

So even the 40 continuity score is not clean prefix quality. The final artifact was contaminated by the baseline fallback. Once fallback became a hard failure, the arm was ineligible regardless of the cost.

An earlier, smaller slice had looked more encouraging:

| Earlier slice | Current | Prefix-direct |
|---|---|---|
| Deterministic continuity | about 30 | 46.67 |
| Cache-adjusted input cost | about 29K | 100.4K |
| Latency | about 63s | 265s |

That was a real signal. It suggested the same-prefix model sometimes held the evolving task state better.

It also cost about 3.4 times as much, took about 4.2 times as long, and produced a malformed first publication.

I could have put that result in a screenshot, declared victory on reasoning continuity, and kept spending tokens until the narrative got cleaner.

That would have been bad evaluation practice.

The exact-save run tested the mechanism more honestly. Once cache creation, retries, publication validity, and fallback were counted, the promising continuity signal did not survive the full route.

## The winner is search + memory

For this system, with this model, under these constraints, the best compaction technique I tested is the boring one:

Start a fresh bounded compaction worker, let it search canonical history for the evidence it needs, require at least one successful retrieval, and validate the final handoff before accepting it.

It is not theoretically pure. It does not preserve the same model prefix. It asks a worker to reconstruct active state through retrieval.

It is also faster, cheaper, easier to bound, easier to recover, and currently better at producing a valid handoff.

The prefix fork remains a useful research seam. I would test it again if one of these changes:

- the provider offers a durable fork or continuation handle without paid priming;
- cache creation can be amortized across many compactions;
- cache lifetime becomes materially longer;
- the model becomes much more reliable at publishing the required schema;
- the handoff can be produced without repair turns;
- the provider exposes a server-side compaction primitive with verifiable prefix reuse.

But none of those is true enough today to justify making it the default.

## Why I stopped before the biggest run

The original plan included a production-scale held-out matrix after the 262K economic boundary, plus final pairwise judge comparisons.

I did not run all of it.

By then the safe prefix candidates had already failed the 1.5x cost and latency gates by huge margins. The exact-save candidate had also regressed deterministic continuity and required fallback. More judge calls could refine the size of the loss. They could not turn 6.9x cost and 9.0x latency into a passing candidate.

I also had about 5% of the Qwen token plan left.

Stopping was part of the experiment.

This is the part of scientific engineering that gets less applause. Once an arm is mathematically disqualified, continuing to spend scarce compute can become a way of avoiding the decision.

The eval and research seams shipped. The production default did not change.

That is a successful outcome.

## The thing I am actually trying to demonstrate

I am looking for work while building Friday, so yes, this article is also a portfolio piece.

But the thing I want it to show is not “I can use a lot of AI.” Plenty of people can burn a heroic number of tokens now.

I want it to show that I can take an attractive systems idea and make it falsifiable.

I can define the state that matters. I can build a benchmark that is not already saturated. I can combine deterministic checks with calibrated semantic judgment. I can instrument provider caches instead of assuming they work. I can trace fallback contamination. I can put cost and latency next to quality. I can fix the infrastructure bugs the experiment exposes.

And I can decline to ship my favorite idea.

That last part is engineering judgment.

The job is not to produce the cleverest architecture.

The job is to produce the best system you can justify with the evidence you have, then leave yourself a clean path to change your mind when the evidence changes.

In this case, the prefix-stable fork was real. The cache reuse was real. The reasoning-continuity hypothesis was plausible. The implementation work improved the harness.

The winner was still search + memory.

That is less magical than the idea I started with.

It is also the decision I trust.