Why Compressing Your AI Agent’s Context Can Raise Your LLM Bill

A coding agent never forgets a log. On every turn, it sends its whole working history back to the model, so the build output it read at step three is still being sent, and paid for, at step forty.

The obvious fix is to compress that history. We built a compressor for this, and the first lesson was that compression can increase the bill. This piece explains why, which design rules fixed it, and what the result saved on 89 real coding tasks.

A coding agent works in a loop: run a command, read the output, think, run the next one. Each turn, the model sees the whole conversation.

In our traces, tool output makes up about half of all prompt bytes; almost all the rest is encrypted reasoning, which cannot be compressed. That tool output is also highly redundant: progress bars redraw, package managers repeat themselves, and test runners print near-identical lines. And about 93% of input tokens are cache reads, billed in our runs at a tenth of the normal price.

That last fact changes the economics.

Prompt caching works on prefixes. If this turn’s prompt starts with exactly the same bytes as the last one, the provider bills those bytes at the cached rate. Change something early in the conversation, and everything after it is billed at full price again.

Here is the arithmetic for one illustrative 100,000-token turn, at GPT-5.5’s list prices of $5.00 per million uncached and $0.50 per million cached input tokens, shown in the chart at the top: $0.08 with no compression, $0.06 compressed by 30% with the cache intact, and $0.35 compressed by 30% with the cache broken.

The last case is the trap. The compressor removed 30% of the tokens and made the turn more than four times as expensive.

So a compressor must produce the same output for the same input on every turn. Two common designs make that hard.

The first is query-aware compression: the compressor reads the user’s latest message to decide what to keep, so the same log compresses differently when the question changes. The second is machine learning under a time budget: a model that stops at a deadline can return different output depending on machine load.

Headroom, an established open-source compressor and the project our engine started from, does both. That trade-off suits a single developer’s desktop. It did not suit a shared gateway serving many teams.

Our compressor, Kompressor, began as a port of Headroom’s algorithmic compressors. We kept the core idea, compress and retrieve, and rebuilt everything else around five rules.

1. Same input, same bytes. No clock, no randomness, no query and no machine learning model. The output depends only on the input, so the cache survives.

2. Remove redundancy, not meaning. Summarization scores well on compression ratio but loses facts. When we ported Headroom’s prose compressor, it removed 6.2 points more text but lost 12.9 points of details the agents later reused. So we de-duplicate instead: the model always keeps at least one real instance of every distinct line.

3. Label every cut and make it recoverable. Each removal becomes a marker such as [… 412 log lines omitted [ref:k_…]; 3 ERROR, 1 WARN]. The model can pass that reference to a retrieval tool and get the original back for an hour.

4. Never worse, fail open. A result that is not smaller is discarded. Any error sends the request through unchanged.

5. Only machine output. System, developer and assistant messages, outputs under about 512 characters, and blocks carrying cache breakpoints are never touched.

The rest is format work. Each tool output is classified and sent to a compressor built for its format: build logs, compiler diagnostics, JSON, diffs, search results, source code and more. A few results from that work:

All of this runs as a single Rust binary at about 0.2 milliseconds per kilobyte of tool output.

We measured on Terminal-Bench 2.1, a public benchmark of 89 command-line tasks (87 ran in every setup), with the Codex CLI agent running GPT-5.5. Each saving below is the difference between real provider bills, including cache discounts, paired task by task. We didn’t price removed tokens at the list rate, which inflates self-reported savings.

That is 22.7% off the bill and 25% fewer input tokens, with no statistically significant change in tasks solved. Headroom saved 8.2% at its defaults and 11.6% at its most aggressive setting.

The savings did more than lower the bill. Because less text went back to the model on each turn, requests got smaller too: the 90th-percentile request fell from 106.9K to 66.5K tokens, and the share of requests over 200K tokens, close to GPT-5.5’s 258K context window, dropped from 1.9% to 0.13%. Most of the money came from the tasks with the longest logs. Machine learning training and evaluation tasks cost 26% less, and building from source costs 28% less; together with data pipelines, these groups accounted for 83% of the total savings.

Compression is not free money everywhere, and the per-task results show it:

Short, cheap tasks swing more with the agent’s path than with its context. Judge compression on the long, log-heavy work, where the money is.

If you run agents at scale and are evaluating any context compressor, five questions will tell you more than its compression ratio:

1. Is the saving measured on the billed cost, with the cache discount included? Tokens removed are not money saved.

2. Is the output deterministic? Feed it the same input twice and compare the bytes.

3. Can the model get the original back? Every cut should be labeled and retrievable.

4. What does it never touch? System and developer prompts and cache breakpoints should be off limits.

5. Where does it lose? A credible benchmark shows the task types where the tool does not help.

Code and full results

Kompressor is open source on GitHub: https://github.com/hystax/optscale-kompressor. The full benchmark report, with per-task results and methodology, is available on our blog.

Disclosure: I work at Hystax, which builds OptScale AI and the open-source Kompressor discussed here, so I think about this professionally — and, yes, with some bias. I’ve tried to keep this piece focused on the problem. The tool is the worked example, and I also cover cases where it helps less.

Why Compressing Your AI Agent’s Context Can Raise Your LLM Bill was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.