Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching

A new study from arXiv finds that query-aware prompt compression, which invalidates prefix caches on every call, can be more expensive than naive caching under realistic hit rates on Anthropic's Sonnet 4.6 API. The researchers propose Cache-Aware Prompt Compression (CAPC), which uses query-agnostic compression with a tier-preserving ratio bound, achieving mean savings of 49% over cache-only and 64% over query-aware compression on LongBench-v2 while maintaining quality within 0.05 of the uncompressed baseline. CAPC also proved cheapest in production tests including a 94k-token enterprise assistant (51.7% cost reduction) and the tau-bench retail benchmark, where query-aware compression was 40.1% more expensive than vanilla.

arXiv:2607.15516v1 Announce Type: new Abstract: Production LLM deployments combine two cost-reduction primitives: prompt caching a discounted rate for re-used token prefixes and prompt compression fewer tokens sent . The compression literature has standardized on query-aware methods that produce a different compressed prefix per query, mechanically invalidating the prefix-strict cache on every call. We characterize this cost empirically on Anthropic's Sonnet 4.6 API and find caching is far from the rho=1.0 ideal the literature assumes: Sonnet's cache has a two-tier architecture with a sharp threshold near 3,500 tokens, below which the hit rate plateaus at rho~0.83 across 30-call sessions. Our cost model predicts, and experiments confirm, that under realistic rho, query-aware compression beats naive caching at high compression ratios r =6 . We propose Cache-Aware Prompt Compression CAPC , pairing query-agnostic compression with explicit cache control plus a tier-preserving ratio bound that prevents over-compression from pushing the cached prefix into the hot tier. CAPC is the cheapest strategy in 16/16 configurations on LongBench-v2, with mean savings of 49% over cache-only, 64% over query-aware compression, and 90% over vanilla, at quality within 0.05 of the uncompressed baseline. We validate CAPC on three production workloads: an enterprise tool-using assistant with a 94k-token schema prefix 51.7% cost reduction at r=3 ; a graphify knowledge-graph RAG pipeline across two codebases 9.3x vs cache-all on FastAPI, 2.4x on httpx ; and the public tau-bench retail benchmark 50 tasks , where CAPC is the cheapest of four strategies with reward exactly equal to vanilla both 36/50, p=1.00 while query-aware compression is the most expensive at +40.1% over vanilla -- the first production confirmation of the crossover model's negative-ROI prediction on a public benchmark.