{"slug": "how-compaction-works-in-pi", "title": "How Compaction Works in Pi", "summary": "Pi, a coding agent from pi.dev, implements compaction to manage LLM context window limits by summarizing older conversation content while preserving recent messages. Compaction triggers automatically when the context nears its limit or manually via the /compact command, and it helps reduce request costs and context rot.", "body_md": "If you've ever had a long coding session in a coding agent like [Pi](https://pi.dev), Claude Code, or Codex, you will have triggered a compaction.\nThat is because large language models (LLMs) have limited [context windows](https://en.wikipedia.org/wiki/Context_window).\n\nThe context window is what the model can \"see\" while producing a response.\n[Transformer architecture](https://en.wikipedia.org/wiki/Transformer_(deep_learning)) limits how much input an LLM can process.\nThe input for a coding agent session includes all the previous messages and tool calls.\nHence LLMs reject requests that exceed the context window.\n\nIn this post, we will discuss when compaction is needed, and how it works in Pi.\nCompaction is also a useful tool for managing the size of the context window size, which both help reduce the cost of LLM requests and reduce [context rot](https://www.trychroma.com/research/context-rot).\n\nWhen working interactively with a coding agent like Pi, the agents and LLM exchange messages.\nEach request to an LLM contains initial context including a system prompt, as well as some additional input.\nThis is typically files loaded into the context such as `AGENTS.md`\n\n, and tool definitions.\n\nA coding agent's first LLM request contains this initial context, along with a first user message.\n\n```\nrequest 1:\n[system][tools][user]\n```\n\nThis starts a turn. The LLM may first return an assistant message containing tool calls. The agent program executes them and sends their results back to the LLM, which can then return another assistant message. The turn is finished when the assistant has completed generating output.\n\n```\nafter request 1:\n[system][tools][user][assistant: tool call][tool result][assistant]\n                     <------------------->     ^        <--------->\n                     returned by LLM           |        returned by LLM\n                                               |\n                                     produced by the agent\n```\n\nWe continue working, and send another message.\n\n```\nrequest 2:\n[system][tools][user][assistant: tool call][tool result][assistant][user]\n                                                                     ^\n                                                               new user message\n```\n\nEach turn expands the conversation.\nEventually, the history exceeds the context limit.\nThe next request then returns an error such as `Request exceeds the maximum size`\n\n.\n\n```\n[system][tools][user][assistant][....][tool result][user]\n                                                      ^\n                                             exceeds context window\n```\n\nWhen we cannot continue with the existing conversation as-is, we have two choices.\n\nIn theory, there are many ways to implement compaction. For example, we can write a deterministic function which keeps some of what is in the conversation and discards the rest. In practice, though, implementations of compaction use an LLM request to summarize the conversation history.\n\nRegardless of the method, after compaction, the context should have been compressed such that we have room for many new messages and tool calls.\n\n```\n[system][tools][compaction result][user]\n                                    ^\n                               new message\n```\n\nLet's look more closely at how Pi specifically [implements compaction](https://pi.dev/docs/latest/compaction#summary-format).\n\nWhen conversations grow too long, Pi uses compaction to summarize older content while preserving recent work.\nCompaction is triggered when the context limit is nearing the total size of the context window.\nIt can also be manually triggered using the `/compact`\n\ncommand.\n\nPi checks for auto-compaction after a turn ends. Until then, each request extends the existing prompt and can reuse its cached prefix. Pi may also compact mid-turn, if it encounters a context overflow error.\n\nWhen compacting, Pi retains some number of recent messages unchanged.\n\n```\nbefore compaction:\n[system + tools][older turns][recent retained messages]\n```\n\nHow many messages are retained vary by session, but it's determined by a [configurable number of tokens](https://pi.dev/docs/latest/compaction#when-it-triggers).\nPi's current default of 20 thousand tokens comes out to roughly 5-20 turns.\n\nAll the messages before this cut point are extracted and serialized, and will be summarixed. To keep the compaction request within the context limit, Pi truncates tool call results in the history to 2,000 characters. If we didn't somehow reduce some of the history, we would already be above the context limit. Tool outputs are a reasonable place to cut because they have a more intermediate nature.\n\nThe compaction request that Pi sends differs from regular conversational requests.\n\nThe result of the compaction is appended to the Pi session as a compaction entry, and the session can now continue. After the compaction request, the context has been compressed.\n\n```\nafter compaction:\n[system][tools][summary][recent turns][new user message]\n```\n\nThere is now room in the conversation context for many more messages.\n\n[Prompt caching](https://earendil.com/posts/prompt-caching) is used by LLM providers to make repeated requests in the same conversation more cost efficient.\nIn an active coding session, we pay less for the context that has already been generated by the model.\nThis caching requires an exact prefix match, so compacting a session will break the prompt cache.\n\n```\ncached before compaction:\n[system][tools][older history][recent retained turns]\n<-------------------- cached prefix -------------------->\n\nfirst request after compaction:\n[system][tools][summary][recent retained turns][new user message]\n<-- reusable -->^\n                |\n        first changed token\n                |\n                +-- everything after this point must be recomputed\n```\n\nThe retained turns contain the same tokens, but they now follow a different prefix. Their previous cached state therefore cannot be reused.\n\nNew requests after compaction will benefit from prompt caching again.\n\nSince Pi is extensible and malleable, you can replace its compaction with your own. To test a different compaction mechanism, ask Pi to create an extension with your own custom compaction prompt.", "url": "https://wpnews.pro/news/how-compaction-works-in-pi", "canonical_source": "https://earendil.com/posts/compaction-in-pi/", "published_at": "2026-08-13 06:30:00+00:00", "updated_at": "2026-08-13 06:38:43.219483+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools"], "entities": ["Pi", "pi.dev", "Claude Code", "Codex", "Chroma"], "alternates": {"html": "https://wpnews.pro/news/how-compaction-works-in-pi", "markdown": "https://wpnews.pro/news/how-compaction-works-in-pi.md", "text": "https://wpnews.pro/news/how-compaction-works-in-pi.txt", "jsonld": "https://wpnews.pro/news/how-compaction-works-in-pi.jsonld"}}