{"slug": "cache-aware-prompt-compression-a-two-tier-cost-model-for-llm-api-caching", "title": "Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching", "summary": "A new study from arXiv finds that query-aware prompt compression, which invalidates prefix caches on every call, can be more expensive than naive caching under realistic hit rates on Anthropic's Sonnet 4.6 API. The researchers propose Cache-Aware Prompt Compression (CAPC), which uses query-agnostic compression with a tier-preserving ratio bound, achieving mean savings of 49% over cache-only and 64% over query-aware compression on LongBench-v2 while maintaining quality within 0.05 of the uncompressed baseline. CAPC also proved cheapest in production tests including a 94k-token enterprise assistant (51.7% cost reduction) and the tau-bench retail benchmark, where query-aware compression was 40.1% more expensive than vanilla.", "body_md": "arXiv:2607.15516v1 Announce Type: new\nAbstract: Production LLM deployments combine two cost-reduction primitives: prompt caching (a discounted rate for re-used token prefixes) and prompt compression (fewer tokens sent). The compression literature has standardized on query-aware methods that produce a different compressed prefix per query, mechanically invalidating the prefix-strict cache on every call. We characterize this cost empirically on Anthropic's Sonnet 4.6 API and find caching is far from the rho=1.0 ideal the literature assumes: Sonnet's cache has a two-tier architecture with a sharp threshold near 3,500 tokens, below which the hit rate plateaus at rho~0.83 across 30-call sessions. Our cost model predicts, and experiments confirm, that under realistic rho, query-aware compression beats naive caching at high compression ratios (r>=6). We propose Cache-Aware Prompt Compression (CAPC), pairing query-agnostic compression with explicit cache_control plus a tier-preserving ratio bound that prevents over-compression from pushing the cached prefix into the hot tier. CAPC is the cheapest strategy in 16/16 configurations on LongBench-v2, with mean savings of 49% over cache-only, 64% over query-aware compression, and 90% over vanilla, at quality within 0.05 of the uncompressed baseline. We validate CAPC on three production workloads: an enterprise tool-using assistant with a 94k-token schema prefix (51.7% cost reduction at r=3); a graphify knowledge-graph RAG pipeline across two codebases (9.3x vs cache-all on FastAPI, 2.4x on httpx); and the public tau-bench retail benchmark (50 tasks), where CAPC is the cheapest of four strategies with reward exactly equal to vanilla (both 36/50, p=1.00) while query-aware compression is the most expensive at +40.1% over vanilla -- the first production confirmation of the crossover model's negative-ROI prediction on a public benchmark.", "url": "https://wpnews.pro/news/cache-aware-prompt-compression-a-two-tier-cost-model-for-llm-api-caching", "canonical_source": "https://arxiv.org/abs/2607.15516", "published_at": "2026-07-20 04:00:00+00:00", "updated_at": "2026-07-20 14:07:50.261290+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-research"], "entities": ["Anthropic", "Sonnet 4.6", "LongBench-v2", "FastAPI", "httpx", "tau-bench"], "alternates": {"html": "https://wpnews.pro/news/cache-aware-prompt-compression-a-two-tier-cost-model-for-llm-api-caching", "markdown": "https://wpnews.pro/news/cache-aware-prompt-compression-a-two-tier-cost-model-for-llm-api-caching.md", "text": "https://wpnews.pro/news/cache-aware-prompt-compression-a-two-tier-cost-model-for-llm-api-caching.txt", "jsonld": "https://wpnews.pro/news/cache-aware-prompt-compression-a-two-tier-cost-model-for-llm-api-caching.jsonld"}}