When it comes to LLMs, context rules everything around us. An agent loop is inherently quadratic: messages accumulate and every iteration you send the full stack of messages to the model. That stack of messages is the context, and it starts with a system prompt, descriptions of the tools, and your instructions. Messages upon messages get added. Tool responses (in other words, the agent reading your source code) usually end up being the majority of the context. This impacts cost (see Expensively Quadratic: the LLM Agent Cost Curve) and quality. You want to give the model just the "right amount" of context, but how?
Compaction is the process of taking your current (long) conversation and rewriting or removing stuff to make it a shorter conversation.
Shelley's approach to compaction is to insert a "nudge" message from the harness at about 200k tokens and to give the model a compaction tool that lets it (a) truncate specific tool outputs and (b) "squish" together sections of history into smaller, rewritten segments (which will then have pointers to the originals). 200k seems like the sweet spot for current frontier models, but we've left the exact number configurable.
The compaction tool requires a few rounds. First, the model gets an "index" so that it can identify the messages by a sequential id. Then, the model suggests messages to collapse or trim. Shelley has a "linter" that verifies that the compaction isn't too big or too little. The model tries again, and usually the second time is just right. After compaction, the conversation continues as it always did, except the squished chunks are shorter, the verbose tool outputs are trimmed, and evidence of the compaction is no longer visible to the LLM.
From a data model perspective, a Shelley conversation has some mutable state (slug name, model choice, current draft) and an append-only log of messages. Compaction is materialized as special "compaction" messages, and the harness, when concatenating all the messages together to send to the LLM, replaces the relevant chunks with their rewrites and trimmed results. If the model ever wants to read something that got compacted away, the data is in SQLite, and Shelley has a skill to hint at the right query. (One of Shelley's principles is the ability to be introspective. I tuned the linter heuristics by having Shelley introspect and backtest its own past conversations.) This is Shelley's third(!) implementation of compaction. The previous two implementations started a new "generation" of a conversation inside the same conversation, with a brand-new context and a summary (first implementation), and preserving the most recent messages (second implementation). This third implementation keeps it going as one conversation, but the history is somewhat rewritten. I'd love to believe this is our last implementation, but model capabilities have been changing quickly, and I expect there will be more iteration.
From the cost perspective, this is the cost graph for a recent long Opus 5.5 conversation. It adds up to about $25 and has gone through 4 compactions, with the context length capping out at about 200k. The little arrows where the price spiked are $1 cache misses, probably when I stepped away from the computer. (You can get this graph in your Shelley session by clicking on the context length counter.) We've been rocking this implementation for a few weeks, and we think it works quite well. We've enabled "auto compaction" by default. I used to click on "Compact and Send" every once in a while and look at the context length. Now, I just talk, and talk, and talk, and it's all good. I've also started letting my agentic conversations meander a bit more, do follow-ups and so on.
Tune in next week to hear about switching to asynchronous tool and subagent calls. Or maybe we'll talk more about multiple input modalities!
If you're interested in more of our writing about coding agents:
- [The Unreasonable Effectiveness of an LLM Agent Loop with Tool Use](https://sketch.dev/blog/agent-loop)
- [Expensively Quadratic: the LLM Agent Cost Curve](https://blog.exe.dev/expensively-quadratic)
- [The Topology of an LLM Agent Loop With Tool Use](https://blog.exe.dev/topology)
P.S.: Here's a behind-the-scenes look at yak-shaving involved in building the animation above. I built it using Shelley, after having earlier built a bespoke program using Excalidraw, and after adding animations to said bespoke drawing program.