Anish Thilagar
Founding Engineer
Alex Siegenfeld
Founding Engineer
Edward Zhang
Founding Engineer
Mitchell Gilmore
Founding Engineer
Preston Zhou
Co-founder, CEO
Eitan Borgnia
Co-founder, COO
As AI costs skyrocket, token leaderboards are being replaced with daily token quotas and companies are frantically trying to reduce spend.
By compacting agent traces after periods of user inactivity, we find you can save 50% on coding workloads. Using a compaction model that runs at 50k+ tok/s allows you to do this without harming the user experience.
Figure 1: The cost over time of a real Fable 5 trace from a feature our CTO worked on in Jacq. Compacting after periods of over 5 mins of inactivity (cache-miss), would have saved 55% on this example.
Relace has always focused on small, specialized models that lower costs and improve UX for coding agents. Launching our own AI coding product, Jacq, has been a goldmine of insights on new strategies.
Our first lesson was that output tokens don't dominate spend anymore. We broke down usage statistics and found cost for long-running conversations is driven by input tokens: early tokens are read from cache hundreds of times, and gaps in usage trigger expensive cache rewrites.
Compaction on cache miss offers major savings on both fronts. Compacting regularly turns cache read costs from quadratic to linear, and doing it on cache-miss dampens the cost of rewriting a large cache.
The only problem is latency. Standard self-summary takes 1-2 minutes, which makes frequent compaction infeasible from a product perspective. That's why we trained Relace Compact to run at well over 50k tok/s — fast and accurate enough to be completely imperceptible to users.
Our model is already publicly available in Jacq, and also accessible via API, so you can integrate it into your own agent!
API Token Pricing Basics
The shift to agentic use cases has created a widespread inference pattern in which users send requests with an accumulating conversation prefix.
Since the attention mechanism in LLMs is causal, KV computation for a given token depends only on tokens that came before it. Instead of recomputing KV pairs for a conversation prefix after each new message, inference providers maintain a short-lived cache to reuse them.
Thus API pricing consists of three categories:
Cache writes on any new tokens sent by the user
Cache reads on conversation prefixes the model has seen before
Output tokens on new tokens the model generates
For Claude Fable 5, the costs are $12.50/million on cache write, $1/million on cache read, and $50/million output tokens. Figure 2: A visual breakdown of how messages in an agent trace are classified for billing purposes.
Note that output tokens must be written to cache after they are produced by the model, so for Fable 5 you effectively pay $62.5/million tokens.
Cache reads: long context blowup #
Agents built on early models experienced degraded performance after 100k tokens, so users were forced to manicure context to retain performance. In those days, cost was dominated by output tokens (we developed Fast Apply to drop cost for output code edits by 3-4x).
Today, agents remain productive well into their 1 million context windows, and users have the luxury of recklessly allowing context to grow.
Figure 3: As the conversation history grows, the cost of a request moves from output-dominated to the read-dominated regime.
If we integrate this linearly increasing input cost over the course of a coding session, we find that cache read costs actually grow quadratically with session length. As a toy example, consider a conversation with a series of tool/user inputs of length (t_{in}) followed by model responses of length (t_{out}). Let (p_{write}), (p_{read}), and (p_{out}) be the different API prices. Then, on the (i)th turns we incur a cost of
$$p_{read} (i-1) (t_{in} + t_{out}) + p_{write} t_{in} + p_{out} t_{out} ~.$$ Summing over an (n) turn session, we get a total session cost of
$$\begin{align*} C &= \sum_{i=1}^{n} \Bigl[ p_{\mathrm{read}}(i-1)(t_{\mathrm{in}}+t_{\mathrm{out}}) + p_{\mathrm{write}}t_{\mathrm{in}} + p_{\mathrm{out}}t_{\mathrm{out}} \Bigr] \ &= \frac{n(n-1)}{2} p_{\mathrm{read}}(t_{\mathrm{in}}+t_{\mathrm{out}}) + np_{\mathrm{write}}t_{\mathrm{in}} + np_{\mathrm{out}}t_{\mathrm{out}} \ &= \Theta(n^{2})p_{\mathrm{read}} + \Theta(n)(p_{\mathrm{write}}+p_{\mathrm{out}}) \end{align*} ~.$$
So despite being almost two orders of magnitude cheaper than output tokens, cache reads already dominate costs in this long context setting. Things get even worse when you consider cache miss events, which happen a lot in practice.
Cache (re)writes #
Memory is expensive, so cached KV pairs are only stored on a provider’s server for a short period of time (TTL). Anthropic, for example, offers a TTL of either 5 minutes or 1 hour, each with its own pricing tier.
Once that window passes, any new message in the session triggers a cache write of the entire conversation prefix at a 10x markup. For an 800k token conversation on Fable 5, a single cache miss costs $10!
Figure 4: When the TTL expires, the cached tokens go stale and need to all be rewritten on the next user message.
With engineers now multitasking on dozens of sessions in parallel, we notice it's becoming increasingly common for sessions to be left idle beyond the TTL in Jacq.
Continuing the example above, assume that every (k) turns we miss the TTL and the cache goes stale. The (j)th time this happens, we incur
$$p_{\mathrm{write}}(jk-1)(t_{\mathrm{in}}+t_{\mathrm{out}})$$ Over the course of (n) total turns this adds up to a total extra cost of
$$\begin{align*} C_{\mathrm{extra}} &= \sum_{j=1}^{m} p_{\mathrm{write}}(jk-1)(t_{\mathrm{in}}+t_{\mathrm{out}}) \ &= \Theta\left(\frac{n^{2}}{k}\right)p_{\mathrm{write}}. \end{align*}$$ Asymptotically then, our cost goes as:
$$C_{\mathrm{total}} = \Theta(n^{2})p_{\mathrm{read}} + \Theta\left(\frac{n^{2}}{k}\right)p_{\mathrm{write}} + \Theta(n)(p_{\mathrm{write}}+p_{\mathrm{out}}).$$ In other words, the total cost of both cache reads and cache writes grows quadratically with the number of turns, while output token costs remain roughly constant per turn and grow only linearly.
Compact on Cache Miss #
Since users don't spend time manicuring context anymore, their conversations often get bloated with stale information like repeated reads of the same file, irrelevant sections of bash outputs, etc. A natural strategy to break the quadratic input token costs is therefore to regularly compact the chat history throughout the coding session.
There are two obstacles to doing this well.
For one, editing previous messages in the conversation ad hoc incurs an expensive cache miss for all subsequent tokens. If you're not careful, this can easily outweigh cache read costs. The best opportunity is on cache miss. Once we’re past the TTL, we’ll have to rewrite the whole prefix anyway, and by compacting, we actually reduce the size of the expensive cache write.
Figure 5: Compacting on cache miss nearly halves the cost of a 50-turn trace. For best results, we recommend keeping the conversation as close to 128k tokens as possible.
The second obstacle is deciding whether information is actually stale enough to prune is hard. You can use a standard LLM, but it takes 1-2 mins for long conversations and isn't cheap.
If you do this right when a user sends a message, it's a horrible user experience. If you do it in the background, you may spend a lot on compacting conversations users never return to.
Relace Compact #
To solve this, we built an absurdly fast compaction model that can prune a long agent trace in seconds. We trigger it whenever a user sends a message beyond TTL, and it actually runs faster than the normal prefill time required for a cache write.
Relace Compact is a modified LLM designed for token-level classification, which means we can run it at 50k tok/s and an order of magnitude lower than cache read cost ($0.20/million input and $0.20/million output tokens). We'll share more details in a subsequent blog post.
To estimate expected savings, we broke down agent traces in Jacq and simulated the change in cost if we had compacted on cache miss (which happens on 1 out every 4 user messages). The histogram of the results is displayed below:
Figure 6: Replaying 15 days of real Jacq traces with compaction on cache-miss to 128k tokens yields an estimated aggregate savings of 56%.
Even a conservative compaction strategy down to 128k tokens saves over 56% on token costs, and after heavily testing the strategy in Jacq, we've confirmed it's imperceptible in both latency and quality.
This is a simple method for a huge win in cost savings! We're continuing to iterate to see how far we can push compaction until we begin to notice a difference in performance.
Conclusion #
For over a year, frontier labs have been heavily discounting models in their coding products relative to API pricing — users who exhaust session limits consume 40 and 70 times the cost of the max subscription for Claude Code and Codex respectively. We argue that going back to the basics of token-based billing — and simply not being wasteful with context — is a strong place to start the optimization process.