{"slug": "i-tested-ai-context-compaction-the-clever-fork-lost", "title": "I Tested AI Context Compaction. The Clever Fork Lost.", "summary": "In a test of AI context compaction techniques, the 'clever' prefix-direct fork approach lost to the current production baseline, search + memory, according to a developer who built an eval to make the baseline fail. The prefix-direct method, which forks an exact cached prefix to reuse the provider's cache, was outperformed by the baseline that searches bounded canonical history and publishes a validated handoff. The developer concluded that architectural elegance does not guarantee practical success, and the baseline remains the winner.", "body_md": "# I Tested AI Context Compaction. The Clever Fork Lost.\n\nPosted August 16, 2026\n\nI had a clever idea about AI context compaction.\n\nIt was actually clever.\n\nIt also lost.\n\nThat distinction matters because the agent world is full of ideas that feel so architecturally satisfying that people skip the annoying step where they prove the idea is worth shipping.\n\nI did not want to do that.\n\nThe idea was this: when an AI conversation gets too large, do not start a fresh compaction request and paste the history into it. Fork the existing conversation at its exact prefix, append one new instruction asking the same model to write a context handoff, and let the provider reuse the cache it already built.\n\nIn theory, the fork gets two benefits at once:\n\n- The model stays inside the same visible working context instead of reading a reconstructed transcript.\n- The provider can reuse the expensive prefix instead of processing it again at full price.\n\nThis felt like the right shape. It preserves continuity. It respects how prompt caches work. It avoids shipping a giant inline transcript to a new worker.\n\nSo I built it.\n\nThen I built an eval designed to make the current approach fail.\n\nThen the current approach won.\n\n## What compaction is actually trying to preserve\n\nLong agent conversations eventually become economically stupid before they become technically impossible.\n\nA model may support hundreds of thousands of tokens, but that does not mean every turn should keep paying to drag the entire history forward. Tool results accumulate. Old branches remain in the transcript. Completed work keeps consuming attention. Latency grows. Cache behavior gets more important. At some point the system needs a smaller handoff document that can carry the work into the next context window.\n\nThe lazy framing is that compaction is summarization.\n\nIt is not.\n\nA useful compaction has to preserve the live state of the work:\n\n- the active objective;\n- decisions and why they were made;\n- evidence supporting or weakening each hypothesis;\n- constraints that still apply;\n- constraints that were superseded;\n- actions already completed;\n- writes that must not be repeated;\n- uncertainty that should remain uncertainty;\n- the next discriminating action.\n\nThat is closer to a transaction log plus a project handoff than a summary.\n\nIf the original conversation says, “We tried A, evidence X disproved it, constraint Y changed, and the next move is B,” a bad compaction remembers A and Y as facts. A good compaction preserves the causal update: A is dead, Y is old, and B is next because of X.\n\nThis is also why I did not ask the benchmark to preserve hidden chain of thought. The model does not expose some pristine internal diary that can be copied into a new request. What the system can preserve and evaluate is provider-visible working state: concise reasoning notes, decisions, evidence, rejected paths, uncertainty, completed actions, and next steps.\n\nThat is the state that matters operationally anyway.\n\n## The techniques\n\nI compared several ways of producing the handoff.\n\n| Technique | How the handoff gets made | What it is testing |\n|---|---|---|\n| Full history | Continue with the entire trajectory | Upper-bound control |\n| Lossy summary | Compress aggressively without recovery | Lower-bound control |\n| Search + memory | A fresh compactor searches bounded canonical history, retrieves evidence, then publishes a validated handoff | Current production baseline |\n| Prefix-direct | Append a handoff instruction to an exact cached prefix and have that model publish directly | Continuity plus cache reuse |\n| Prefix-plan-worker | The exact prefix fork writes a structured brief, then a fresh worker searches history and publishes the final handoff | Same-context planning plus controlled publication |\n| Sandbox-tool fork | Let the prefix fork execute deterministic disposable tools | Whether tool access is the missing causal ingredient |\n\nThe phrase “exact cached prefix” is doing a lot of work here.\n\nIt is not enough for the logical Friday transcript to look the same. The provider-facing bytes have to remain the same too. System instructions, tool schemas, cache markers, message ordering, tool-call serialization, and repair turns can all change the wire representation.\n\nI added adapter-level proofs for that. The active request had to be a byte-identical prefix of the compaction request. Repair turns had to extend the fork without rewriting the inherited source prefix. Even a compaction tool call could not be allowed to displace one of Qwen's explicit cache markers.\n\nThat work found real bugs. The first implementation preserved the application-level prompt hash while moving provider cache boundaries. It looked exact from inside the harness and was not exact on the wire.\n\nThis is why “we use prompt caching” is not evidence. Cache behavior is a provider contract, not a vibe.\n\n## The hypothesis\n\nMy favored hypothesis was:\n\nA byte-identical prefix fork will preserve more reasoning continuity than a fresh search-based compactor while staying within 1.5 times the baseline cost and latency because most of the inherited context will be cached.\n\nI gave it real ship gates:\n\n| Gate | Requirement |\n|---|---|\n| Continuity improvement | At least 15 points over current |\n| Confidence | Paired 95% interval above zero |\n| Pairwise judge preference | At least 60% of non-ties |\n| Deterministic final-state success | No regression |\n| Cache-adjusted cost | No more than 1.5x current |\n| p95 compaction latency | No more than 1.5x current |\n| Exact inherited prefix | 100% |\n| Safety and duplicate writes | Zero failures |\n| Prefix fallback | Zero |\n\nThat last gate became important later.\n\nIf a prefix experiment silently falls back to the existing search compactor, it can inherit the baseline's handoff quality while still being labeled a prefix success. That is benchmark contamination. The eval now records prefix fallbacks explicitly and makes any one of them a hard failure.\n\n## I needed an eval that was not already solved\n\nNeedle-in-a-haystack tests are useful, but they were too easy for this question. A compaction can retain a literal fact and still destroy the work.\n\nI borrowed the shape of the benchmark from several stronger eval traditions:\n\n[LongMemEval-V2](https://xiaowu0162.github.io/longmemeval-v2/)for dynamic state, workflow knowledge, environment gotchas, and premise awareness.[NoLiMa](https://github.com/adobe-research/NoLiMa)for minimizing lexical overlap so retrieval requires latent association instead of keyword matching.[tau2-bench](https://github.com/sierra-research/tau2-bench/blob/main/docs/evaluation.md)for checking the exact final state rather than demanding one privileged path.[RULER](https://github.com/NVIDIA/RULER)and[LongBench v2](https://arxiv.org/abs/2412.15204)for configurable long-context complexity and reasoning rather than raw recall.[MT-Bench's judge study](https://arxiv.org/abs/2306.05685)for reference-guided judging and swapped answer order to expose position bias.\n\nThe resulting benchmark has five seeded families.\n\n| Family | The reasoning state that must survive |\n|---|---|\n| Incident diagnosis | Evidence reverses an early hypothesis; choose the next discriminating test |\n| Changing implementation plan | Preserve revised dependencies and do not resurrect a superseded design |\n| Data reconciliation | Derive a source-precedence policy, absorb corrections, apply it to a paraphrased case |\n| Interrupted tool workflow | Resume completed work, avoid duplicate writes, survive an environment gotcha |\n| Product judgment | Preserve the user's rationale, taste, trust boundaries, and scope on a new proposal |\n\nEach fixture carries a gold `ReasoningStateGraphV1`\n\n. It names the active goal, hypotheses, evidence edges, active and superseded constraints, decisions, rationale, completed actions, unresolved uncertainty, forbidden repetitions, and expected next action.\n\nThe post-compaction challenge is not “what was the secret code from turn 40?”\n\nIt is closer to:\n\nThe deployment symptom changed in this specific way. Given the evidence already collected, what should you test next, what should you not repeat, and which earlier hypothesis is now superseded?\n\nThat is much harder to bluff.\n\n## Deterministic scoring first, model judging second\n\nI wanted most of the score to come from things a script could verify.\n\nThe composite was designed as:\n\n```\ncontinuity score = 60% deterministic outcome/state + 40% semantic judge\n```\n\nThe deterministic side checks exact sandbox state, completed-action preservation, forbidden duplicate writes, active versus superseded constraints, and required next actions.\n\nThe semantic judge sees a bounded packet containing the gold state graph, the challenge, the handoff, and the continuation result. It does not see the arm name or cost. It must cite gold graph identifiers for every finding. Pairwise comparisons run in both answer orders; inconsistent preferences become ties.\n\nBefore the judge's scores count, it has to pass known corruptions: omitted rationale, resurrected constraints, false certainty, verbosity attacks, and identical-answer ties.\n\nI do not think “LLM as a judge” is automatically scientific. I think a calibrated, blinded, bounded judge can cover semantic properties that are expensive to reduce to string matching, as long as deterministic state remains the authority where deterministic state exists.\n\n## Calibration: prove the benchmark has room\n\nBefore comparing compaction techniques, I ran a 64K calibration.\n\n| Control | Aggregate continuity | What it should prove |\n|---|---|---|\n| Full-history oracle | 97.32 | The tasks are solvable |\n| Search + memory | 74.84 | The baseline is useful but not saturated |\n| Deliberately lossy summary | 9.07 | The benchmark detects destroyed reasoning state |\n\nNo baseline family exceeded 84.4. The oracle's weakest family still scored 94.4. The judge calibration passed 6 out of 6 adversarial cases.\n\nThat was the green light.\n\nThe eval was not just measuring whether the model could do the task at all. It also was not an easy suite where every halfway competent compactor scored 98.\n\nThere was room for the clever idea to win.\n\n## The cache part actually worked\n\nQwen's explicit context cache is real. Its [official context-cache documentation](https://docs.qwencloud.com/developer-guides/text-generation/context-cache) lets callers mark stable content with `cache_control`\n\n. The published economics at the time of this experiment priced explicit writes at 125% of normal input and reads at 10%, with a short renewable lifetime.\n\nIn a direct repeated-prefix probe, the second request contained 5,438 input tokens and Qwen reported 5,432 cached tokens.\n\nThat is an excellent cache hit.\n\nIt is also not the whole economic story.\n\nQwen does not expose a free “save this prefix for later” endpoint. Creating the explicit cache requires a model request. The first pass pays the cache-creation premium. The compactor still has to generate and sometimes repair the handoff. If the attempt fails and falls back, the system pays for both paths.\n\nCache-adjusted cost is therefore closer to:\n\n```\nordinary input\n+ 1.25 * cache creation\n+ 0.10 * cache reads\n+ output and reasoning\n+ retries\n+ fallback\n```\n\nThe cache hit can be nearly perfect while the product route is still expensive.\n\nThat is exactly what happened.\n\n## The result\n\nThe cleanest exact-save comparison looked like this:\n\n| Measure | Search + memory | Exact prefix fork | Fork vs. baseline |\n|---|---|---|---|\n| Deterministic continuity | 60 |\n40 | -20 points |\n| Input tokens | 18,254 | 224,686 | 12.3x |\n| Cached input tokens | 3,695 | 133,431 | 36.1x |\n| Cache creation tokens | 14,541 | 91,219 | 6.3x |\n| Cache-adjusted input cost | 18,563.75 |\n127,402.85 | 6.9x |\n| Compaction latency | 75.7s |\n682.3s | 9.0x |\n| Exact inherited prefix | Not applicable | 2 of 2 | Passed |\n| Prefix fallback | 0 | 1 | Hard failure |\n\nThe fork did the technically impressive thing. It preserved the exact inherited prefix. It produced large cache reads. It proved the provider adapter could keep its markers stable across an appended compaction request and a repair turn.\n\nThen it failed to publish a valid handoff in its two allowed attempts and fell back to search.\n\nSo even the 40 continuity score is not clean prefix quality. The final artifact was contaminated by the baseline fallback. Once fallback became a hard failure, the arm was ineligible regardless of the cost.\n\nAn earlier, smaller slice had looked more encouraging:\n\n| Earlier slice | Current | Prefix-direct |\n|---|---|---|\n| Deterministic continuity | about 30 | 46.67 |\n| Cache-adjusted input cost | about 29K | 100.4K |\n| Latency | about 63s | 265s |\n\nThat was a real signal. It suggested the same-prefix model sometimes held the evolving task state better.\n\nIt also cost about 3.4 times as much, took about 4.2 times as long, and produced a malformed first publication.\n\nI could have put that result in a screenshot, declared victory on reasoning continuity, and kept spending tokens until the narrative got cleaner.\n\nThat would have been bad evaluation practice.\n\nThe exact-save run tested the mechanism more honestly. Once cache creation, retries, publication validity, and fallback were counted, the promising continuity signal did not survive the full route.\n\n## The winner is search + memory\n\nFor this system, with this model, under these constraints, the best compaction technique I tested is the boring one:\n\nStart a fresh bounded compaction worker, let it search canonical history for the evidence it needs, require at least one successful retrieval, and validate the final handoff before accepting it.\n\nIt is not theoretically pure. It does not preserve the same model prefix. It asks a worker to reconstruct active state through retrieval.\n\nIt is also faster, cheaper, easier to bound, easier to recover, and currently better at producing a valid handoff.\n\nThe prefix fork remains a useful research seam. I would test it again if one of these changes:\n\n- the provider offers a durable fork or continuation handle without paid priming;\n- cache creation can be amortized across many compactions;\n- cache lifetime becomes materially longer;\n- the model becomes much more reliable at publishing the required schema;\n- the handoff can be produced without repair turns;\n- the provider exposes a server-side compaction primitive with verifiable prefix reuse.\n\nBut none of those is true enough today to justify making it the default.\n\n## Why I stopped before the biggest run\n\nThe original plan included a production-scale held-out matrix after the 262K economic boundary, plus final pairwise judge comparisons.\n\nI did not run all of it.\n\nBy then the safe prefix candidates had already failed the 1.5x cost and latency gates by huge margins. The exact-save candidate had also regressed deterministic continuity and required fallback. More judge calls could refine the size of the loss. They could not turn 6.9x cost and 9.0x latency into a passing candidate.\n\nI also had about 5% of the Qwen token plan left.\n\nStopping was part of the experiment.\n\nThis is the part of scientific engineering that gets less applause. Once an arm is mathematically disqualified, continuing to spend scarce compute can become a way of avoiding the decision.\n\nThe eval and research seams shipped. The production default did not change.\n\nThat is a successful outcome.\n\n## The thing I am actually trying to demonstrate\n\nI am looking for work while building Friday, so yes, this article is also a portfolio piece.\n\nBut the thing I want it to show is not “I can use a lot of AI.” Plenty of people can burn a heroic number of tokens now.\n\nI want it to show that I can take an attractive systems idea and make it falsifiable.\n\nI can define the state that matters. I can build a benchmark that is not already saturated. I can combine deterministic checks with calibrated semantic judgment. I can instrument provider caches instead of assuming they work. I can trace fallback contamination. I can put cost and latency next to quality. I can fix the infrastructure bugs the experiment exposes.\n\nAnd I can decline to ship my favorite idea.\n\nThat last part is engineering judgment.\n\nThe job is not to produce the cleverest architecture.\n\nThe job is to produce the best system you can justify with the evidence you have, then leave yourself a clean path to change your mind when the evidence changes.\n\nIn this case, the prefix-stable fork was real. The cache reuse was real. The reasoning-continuity hypothesis was plausible. The implementation work improved the harness.\n\nThe winner was still search + memory.\n\nThat is less magical than the idea I started with.\n\nIt is also the decision I trust.", "url": "https://wpnews.pro/news/i-tested-ai-context-compaction-the-clever-fork-lost", "canonical_source": "https://rico.codes/ai-context-compaction-evals", "published_at": "2026-08-16 08:22:00+00:00", "updated_at": "2026-08-16 08:41:07.051818+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/i-tested-ai-context-compaction-the-clever-fork-lost", "markdown": "https://wpnews.pro/news/i-tested-ai-context-compaction-the-clever-fork-lost.md", "text": "https://wpnews.pro/news/i-tested-ai-context-compaction-the-clever-fork-lost.txt", "jsonld": "https://wpnews.pro/news/i-tested-ai-context-compaction-the-clever-fork-lost.jsonld"}}