I noticed something weird. Every time my AI agent's process restarted, it would compress the conversation again — even though the last five compressions had been useless.
The prompt was already clean. The system prompt and tool schemas added up to 30K tokens of incompressible floor. Shrinking the message history wasn't going to help. But the agent didn't know that. It ran the compaction loop every single time, burning tokens and time for zero benefit.
The root cause? A single in-memory counter that disappeared on restart.
Context compression is how long-running AI agent conversations stay under the model's context window. You have a threshold — say 50K tokens. When the provider reports the prompt approaching that limit, you compact: summarize old messages into a single system-level entry, drop the full history, and keep rolling.
Simple, right?
But there's a corner case: when the system prompt and tool schemas alone are already close to the threshold, compaction can't help. The message history shrinks, sure, but the total prompt stays over the line. So the next turn triggers compaction again. And again. And again.
To stop this, I added a thrashing guard — a counter that ticks up each time compaction runs but doesn't clear the threshold. After two ineffective rounds, it blocks further compression. Clean.
class CompressionState:
def __init__(self):
self._ineffective_compression_count = 0
def update_from_response(self, usage):
"""Called after each API response with real token counts."""
if self._verify_compaction_cleared_threshold:
if self.last_prompt_tokens >= self.threshold_tokens:
self._ineffective_compression_count += 1
else:
self._ineffective_compression_count = 0
def should_compress(self):
"""Gate check — blocks compression after 2 strikes."""
return self._ineffective_compression_count < 2
The guard works perfectly in a single session. But it lives in memory. When the process restarts — deployment, crash recovery, even a graceful restart — the counter resets to zero. The guard disarms. And the agent compresses the already-compacted conversation one more time.
The pattern was already in the codebase for two other counters: the compression failure cooldown (which prevents retrying a failing provider) and the fallback streak (which tracks how many deterministic fallback summaries were inserted in a row). Both persisted through a durable session-state channel backed by SQLite.
I applied the same pattern to the anti-thrashing counter. Three pieces:
1. A database column. I added compression_ineffective_count
to the sessions table, with accessor methods that return the value or write it back.
def get_compression_ineffective_count(self, session_id: str) -> int:
row = self._conn.execute(
"SELECT compression_ineffective_count FROM sessions WHERE session_id = ?",
(session_id,)
).fetchone()
return row[0] if row else 0
def set_compression_ineffective_count(self, session_id: str, count: int) -> None:
self._conn.execute(
"UPDATE sessions SET compression_ineffective_count = ? WHERE session_id = ?",
(count, session_id)
)
2. A centralized verdict recorder. Every time the update_from_response()
method decides whether the last compaction was effective or not, it routes through _record_ineffective_compression_verdict()
. This method updates both the in-memory counter and the database row — atomically, in the same code path.
def _record_ineffective_compression_verdict(self, was_ineffective: bool):
if was_ineffective:
self._ineffective_compression_count += 1
else:
self._ineffective_compression_count = 0
setter = getattr(self._session_db, "set_compression_ineffective_count", None)
if callable(setter):
try:
setter(self._session_id, self._ineffective_compression_count)
except Exception as exc:
logger.debug("persist ineffective count failed: %s", exc)
3. Load on bind. When the compressor binds to a resumed session, it reads the persisted counter back into memory.
def bind_session_state(self, session_db, session_id):
self._session_db = session_db
self._session_id = session_id
self._ineffective_compression_count = 0 # fallback
getter = getattr(session_db, "get_compression_ineffective_count", None)
if callable(getter):
try:
stored = getter(session_id)
self._ineffective_compression_count = max(0, int(stored))
except Exception:
pass
That's it. The bug only existed because the codebase had two different patterns — one durable (cooldown, fallback streak) and one in-memory (anti-thrashing counter). The fix wasn't new architecture. It was making the third counter use the same durable channel as the first two.
The key insight is that a session is a durable entity, but its compression state was ephemeral. Every other part of the session — messages, config, metadata — lives in SQLite. The anti-thrashing counter was the odd one out.
By moving it into the same persistent channel, three things happen automatically:
The reset semantics didn't change, either. Any real provider response that reads below the threshold still clears the counter — and now it clears durably.
Don't persist no-change verdicts. If the counter didn't change — say, a response came in but _verify_compaction_cleared_threshold
was False (meaning no compaction happened this cycle) — skip the DB write. Two reasons: it's wasted I/O, and it creates a false positive "someone touched this" signal in the session row's modification time.
Mind the null-coalescing. A session row created before the column existed will return None
, not 0
. Always max(0, int(stored))
to avoid TypeError
on a fresh None
.
The guard must check the in-memory value first, then refresh. should_compress()
runs on the hot path — every turn start. The in-memory copy covers the fast path. If it says "blocked", use the durable fallback and skip the DB read. Only refresh from the database when the in-memory copy is zero (disarmed) and you need to prove another agent didn't arm it since bind_session_state()
.
Don't remove the in-memory copy just because the DB is durable. Writing to SQLite on every response is fine (it's a single-row UPDATE on an indexed primary key), but reading from it on should_compress()
when you already have the value in RAM is wasteful. Keep both, write through, read from RAM.
Have you ever found a bug that was just "this value is in memory but everything else is in the database"? I'd love to hear what patterns you use to prevent state fragmentation in agent systems — especially if you're dealing with context windows and compression. Drop your story in the comments.