Context compaction is safe or unsafe only against a specific proposed action. Not in general. The compactor decides what your agent forgets before it knows what your agent will do, so at compaction time "did this lose anything important?" has no answer yet. The lie, if there is one, comes into existence later, when the agent proposes to act.
That sounds like philosophy. It has an exit code.
AI disclosure.I wrotecompaction_omission_gate.py
with an AI assistant and ran it myself: Python 3.13.5, offline, standard library only, no network, no keys, no funds. Every verdict, exit code and hash below is pasted from a real local run. I ran the whole demo twice and the twooutput.txt
files are byte-for-byte identical (sha256 2f0ae2c6d54b1d35caefb3da12af3a4189aff05e5d8c59321b63150f27eef376
). The fixtures are synthetic and mine. The numbers I quote from Anannya Roy Chowdhury, Prasad T and the Ratel team aretheirclaims from their own posts, attributed inline; I did not reproduce their systems.
In short:
kept-sha256=3bfe38c85fcf
on every row), same 36 dropped, and the same rule missing from all four: amount 5000
passes, amount 5001
blocks. Same words, one unit apart.Anannya Roy Chowdhury published My Multi-Agent AI Cost $1,847 in One Weekend on July 16. Her numbers: $1,847 burned, 82% cut, context compressed "from 12,000 tokens to 340." That is 97% of the context gone. I asked her in the comments whether she'd found a way to catch a bad compression
Her reply named the hole better than my question did:
"A bad compressor can absolutely 'lie' by omission, which is arguably more dangerous than naive replay because
the failure is silent."
She called it the core challenge of her Part 2, which was not out when I wrote this. So this post is not a rebuttal of anything. It is an attempt to answer the question she agreed was open.
Two other people hit the same wall in the same week. Prasad T's Teaching a Qwen agent to forget put a
superseded_by
link on memory records so a contradicted fact is dropped from recall but kept in an audit trail. The line I keep coming back to, from the thread under his post: forgetting you can't inspect is just data loss. And the My first design was the obvious one: run the decision on the full context, run it on the compacted context, and if the decision changes, BLOCK. Compare against the truth.
Prasad's post killed it. His FAMA measurement (his numbers, not mine) reports recency-only = 0.64 and never-forgets = 0.64. Identical. Never-forgets is the full context. So the "truth" I wanted to compare against is itself wrong 36% of the time. An oracle that lies in a third of cases is not an oracle, and any reader who has read his post gets to close my tab in one line.
Those same two numbers kill a second thing, which is the framing I would have reached for by reflex: compaction hurts because you lose volume. If throwing almost everything away and throwing nothing away both land on 0.64, volume is not the variable. Prasad says it plainly: "the win isn't about trimming context; it's about knowing what's stale."
So the gate has no oracle. It never asks what the model would have decided. It resolves your declared predicates against your stored records. That is a structural fact, not a prediction.
One closed question: among the records the compactor dropped, is there one that binds this action?
Input is the full pre-compaction context, the ids the compactor kept, and a concrete action. The dropped set is computed, never supplied. I don't trust anyone's "here's what I removed" list. Records are typed: policy
(scope, predicate, effect), fact
(field, value, optional supersedes
), chatter
. Ordering is a declared seq
, never a wall clock, so the verdict can't drift with the time of day.
The whole resolution step is this small:
def resolve(field, params, facts):
"""action.params wins; otherwise the newest stored fact for that field."""
if field in params:
return True, params[field]
if field in facts:
return True, facts[field]["value"]
return False, None
If a field resolves nowhere, the gate fails closed rather than assuming harmless. There is no scoring anywhere in the verdict path. PASS, BLOCK, ERROR. No percentages. That's deliberate: on July 16 a Show HN called ReasonGate shipped a "gate" built on regexes, and Simon Willison took it apart in the thread within the hour. Recall of "76-96%... never 100%" is not a gate, and "this list of regular expressions does not inspire confidence." A gate that guesses is a filter with good PR.
Here is the claim as an experiment, and it needs nothing to compare against. One compaction. Four actions.
The fixture is a 52-record support thread. The refund request opens it, the resolution closes it, and the middle carries the boring stuff: internal notes, a tier update, and one dry line at seq 11
: Refunds over 5,000 need a human approver.
The compaction is head+tail, keep the first 8 and the last 8, which is the shape most windowing schemes converge on. That rule sits at position 11 of 52. It is dropped. In all four rows.
| Proposed action | Verdict | Exit |
|---|---|---|
refunds.create {amount: 8400} |
||
BLOCK dropped-binding-constraint |
||
| 1 | ||
refunds.create {amount: 120} |
||
| PASS | 0 | |
refunds.create {amount: 5000} |
||
| PASS | 0 | |
notes.append {ticket: 7741} |
||
| PASS | 0 |
"Same compaction" is a claim, and claims about counts are cheap, so the tool prints a fingerprint of the kept set:
compaction: 52 records, 16 kept, 36 dropped (retention=given, kept-sha256=3bfe38c85fcf)
That digest is identical on all four rows. Same 52 records, same 16 kept, record for record, same 36 dropped. One blocks. Nothing about the compaction moved, and the answer to "is this compaction safe?" moved anyway. That is the whole thesis, and there is no baseline in it, because the claim isn't that some other tool does worse. The claim is that the question is malformed until you name the action.
The 120
row prints a line I like:
note : permissive: dropped policy 'f_020' (allow) resolves TRUE here (amount = 120 lt 500)
but losing a permission cannot license an action; not a block reason
A dropped allow
rule is not danger. It costs you an auto-approval, nothing more. The gate says so instead of blocking on principle.
"A good compactor" is not a thing you can have. There is only "this compaction is safe for that action."
The 5000
row is the one worth poking at. 5000 gt 5000
is FALSE, so the rule wouldn't have fired, so its absence changes nothing. But how would you know the gate computed that rather than pattern-matched its way there? Walk the boundary:
amount |
Verdict |
|---|---|
| 4999 | PASS |
| 5000 | PASS |
| 5001 | BLOCK dropped-binding-constraint
|
| 8400 | BLOCK dropped-binding-constraint
|
5000
and 5001
are the same word to any tokenizer worth the name. The verdicts differ because the gate resolved amount
out of the action's params and did arithmetic against a stored predicate. Lexical similarity has no way to know that 8400 > 5000
, because that is not a fact about words.
Two days ago I asked publicly whether you can catch a bad compression before the turn runs. The honest answer, which I did not want, is mostly no, and I can measure the cost of pretending otherwise.
--precheck
takes the action away from the verdict and asks the general question instead: could any dropped record bind any reachable call of these tools?
$ python3 compaction_omission_gate.py --precheck --retention given fixtures/fx1_refund_thread.json
action : (the verdict below never consults it -- that is the experiment)
compaction: 52 records, 16 kept, 36 dropped (retention=given, kept-sha256=3bfe38c85fcf)
tools : refunds.create
verdict : BLOCK (action-blind) exit 1
reason : may-bind-some-action: dropped policy 'f_011' scopes to refunds.create; some
reachable call of that tool satisfies (amount gt 5000). Which one? Unknown
here: no action was supplied
Run it against each of the four actions and you get the same line four times, because it never reads them. The gate blocks 1 of 4. Precheck blocks 4 of 4. Three of those four blocks are false, and they are not fixable by being cleverer: nothing recovers 120 > 5000 = FALSE
without the 120.
So the control point sits one step later than I hoped, and one step earlier than it costs money. The turn is already spent: the model has read the mutilated context and produced a proposal. Nothing has executed. That's the line my pre-execution gate has always drawn: before the action, not before the thinking. Reading the dropped records to check them costs zero tokens, since compaction governs what gets sent to the model, not what's on your disk. (Money is a different post: the re-billing curve is here.)
Here's the bug my reviewer found in my own tool, and it is the mirror of the one I was proud of.
I had a rule: losing a permission cannot license an action. True, and the gate leans on it. A dropped allow
never blocks. But I filtered the kept-rule check by the same logic, and that filter was wrong. A kept allow
rule is a permission you still have. If the compactor drops the value that permission reads, the permission fires on a value you already took back:
$ python3 compaction_omission_gate.py --retention given fixtures/fx6_stale_permit.json
verdict : BLOCK (dropped-superseder-stale-permit) exit 1
reason : dropped-superseder-stale-permit: kept policy 'f_003' (allow) reads
'customer.tier'. The compactor kept 'f_002' (customer.tier = gold, seq 2) and
dropped 'f_011' (customer.tier = standard, seq 11) and declares supersedes
'f_002'. The rule's own predicate (customer.tier eq gold) is TRUE on the kept
value and FALSE on the newest one: the agent will license this call with a
value you already retracted. Losing a permission cannot license an action, but
KEEPING one while its input is retracted can
Before the fix, that fixture returned PASS exit 0. Silent. A 4,000 refund auto-approved on a gold
tier that was revoked twenty records ago, and my gate, whose entire pitch is "the failure is silent, so catch it," said nothing. I had written the general principle and then applied it to the wrong noun.
Fixing it forced a distinction I had been sloppy about. A stale value has two directions and they are not the same event:
The second one is a real fixture too (fx7
), and the gate declines to block it:
note : stale value, safe direction: kept policy 'f_003' (deny) reads 'customer.tier'.
[...] The rule's own predicate (customer.tier eq restricted) is TRUE on the kept
value and FALSE on the newest one. This compaction makes the agent more cautious
than your records warrant: a cost, not a block reason
A gate that blocks whenever anything changed is just an alarm wired to the whole house. The direction is the point.
Same fixtures, same k
, two action-independent schemes:
| fixture | head_tail | recency |
|---|---|---|
| refund thread (rule in the middle) | BLOCK binding | BLOCK binding |
| superseded fact | BLOCK stale-kept | BLOCK binding |
| genuinely safe | PASS | PASS |
retracted tier, kept allow rule |
||
| BLOCK stale-permit | PASS | |
retracted tier, kept deny rule |
||
| PASS | PASS |
I predicted recency would survive the supersession fixture. It doesn't. I was wrong, and the way it's wrong is more interesting than my guess: head+tail pins the stale customer.tier = standard
at seq 2
and drops the correction at seq 20
, so the agent applies a kept rule to a retracted value. Recency keeps the correction and drops the rule itself. Two popular schemes, opposite failures, same input. "Switch to recency" doesn't fix the failure, it relocates it.
I won't tell you no correct scheme exists; five synthetic fixtures can't carry that. The structural argument is stronger than the table anyway. Safety depends on the action, which the freeze-and-vary run above shows without appealing to any scheme at all. Neither of these schemes can see the action. So neither can be safe in general, and the row where recency wins is luck about where the record sat, not virtue.
Probably, sometimes. And this is where I have to report a section that isn't here.
An earlier draft of this post had a better headline and a fifth act: my gate against a relevance-based compactor, IDF-weighted cosine, keep-top-k, the state of the art, confidently certifying a compaction that dropped a binding rule. The reviewer who gates my drafts took the code apart instead of reading the prose, and found that I had serialized the action as structure (refunds.create amount 8400 currency USD customer c_889
) while indexing every record as prose (Refunds over 5,000 need a human approver.
). Same JSON on both sides. My gate read six fields off that record. My baseline read one.
Whatever that comparison showed, it was not a fact about relevance ranking. It was a fact about what I handed each side. So the ranker is gone, all ninety-odd lines of tokenizer and cosine that existed to lose a fight I had arranged. The thesis never needed it, which I'd have noticed sooner if the number hadn't been flattering.
The honest version of the question is harder. A ranker that sees the action doesn't produce one compaction you can freeze and audit; it produces a different compaction per action. That may well be the right engineering. It also means there is no fixed artifact to check, and "is this compaction safe" stops being a question you can ask once and cache. I don't have a clean answer for that shape. If you do, I'd like to hear it.
One fixture is rolling summarization output: the middle of the thread replaced by eight prose blobs typed as summary
. The gate doesn't recognize that kind, can prove nothing about it, and blocks with dropped-unevaluable-record
. Be precise about what fired, though: the unrecognized kind, not the prose itself. Retype those same eight blobs as chatter
— a known kind the gate reads as your own declaration that a record is throwaway — and it believes you, drops them silently, and returns PASS. I checked; the verdict flips. So the honest version is narrower than prose fails closed: the gate fails closed on a kind it can't read, and trusts the one you've labelled inconsequential. Same knife as the schema limit further down — it holds you to your typing. There's an --allow-unevaluable
flag that turns the unrecognized-kind block into PASS exit 0 with [--allow-unevaluable: you chose to fly blind]
in the report. Not a fix. A signed decision.
This is the uncomfortable one, because rolling summarization is exactly what my own context-tax post prescribes as the cure. I'm auditing my own prescription. If your memory is a pile of summarized text, this gate has nothing to work with, and neither does anything else. That's not a gap in the gate. It's a gap in the store. Prasad's superseded_by
link is the cheapest version of the fix.
mandate-freshness-gate expires
checkpoint-skip-gate
agent-memory-tax-and-backdoor
sliding-window-spend-guard
Drop the tool and the fixtures in a directory and run bash run_demo.sh
. Three stdlib imports (json
, hashlib
, sys
), no network, no keys, no model. The full sweep ends 2 PASS, 4 BLOCK, 1 ERROR -> overall exit 1
, malformed input exits 2 and never 0, and every report ends with a sha256 of its own body. Run it twice and diff. If it isn't byte-identical, I'd want to know.
Every command in the demo passes --retention given
explicitly even though it's the default, because an earlier version of that script quietly recomputed the compaction per action while the prose above it claimed the compaction was frozen. Two of the numbers I'd written were false and I hadn't run the command my own paragraph described. Spelling the flag out makes the sentence and the command the same object.
The schema is the price of admission. policy
needs a scope, a predicate and an effect; fact
needs a field, a value and ideally a supersedes
. If your ingestion layer labels a binding rule as chatter, the gate believes it, and nothing saves you. It holds you to your own typing. That's the honest limit.
So: if compaction safety is a property of the pair rather than of the compactor, where does this check belong in your harness: inside the compactor, in the store, or at the execution gate? I lean toward the execution gate, because it's the only place that knows the action, but that's the place least likely to have the dropped records in hand. And what do you do when your context is prose with no schema? Pay to type it, or fly blind and admit it?
Follow for the numbers from the next gate I build and break. If you've watched an agent lose an instruction to auto-compact and only find out from the outcome, tell me what the dropped record was. I read every comment.