Learning Agent Execution for KV-Cache Management in Agentic Serving

CacheScout, a new KV-cache runtime layer for multi-agent LLM serving developed by researchers, improves KV-cache hit rate by 10-18 percentage points, reduces mean time-to-first-token (TTFT) by 18-45%, lowers mean per-turn latency by 29-38%, and increases peak throughput by up to 57% across real-world multi-agent workloads. Implemented on top of vLLM, CacheScout learns agent execution transitions online to guide cache eviction and proactive prefetching, avoiding the need for predefined workflow graphs or offline training.

arXiv:2608.14624v1 Announce Type: new Abstract: Multi-agent LLM systems have emerged as an important deployment paradigm for AI services, where each user request is decomposed into a sequence of specialized agents. Across these workflows, every agent repeatedly executes a fixed context consisting of system prompts, tool definitions, and few-shot examples, creating substantial opportunities for KV-cache reuse. Existing LLM serving systems, however, manage KV-cache reactively using prefix caching and recency-based replacement, causing reusable agent contexts to be evicted before their next invocation and forcing repeated recomputation. We present CacheScout, an agent-aware KV-cache runtime layer for multi-agent LLM serving. The key insight is that future KV-cache reuse is governed by agent execution semantics rather than cache recency alone. CacheScout captures these semantics by learning agent execution transitions online, without requiring predefined workflow graphs or offline training, and uses the learned execution model to guide both cache eviction and proactive prefetching while leaving the serving critical path unchanged. We implement CacheScout on top of vLLM. Across representative real-world multi-agent workloads, CacheScout improves KV-cache hit rate by 10-18 percentage points, reduces mean TTFT by 18-45%, lowers mean per-turn latency by 29-38%, and increases peak throughput by up to 57%. These benefits also generalize to larger models, reducing TTFT by up to 54% while sustaining 37% higher throughput.