{"slug": "lru-is-harder-to-beat-than-the-kv-cache-papers-suggest", "title": "LRU is harder to beat than the KV-cache papers suggest", "summary": "A prefix-cache simulator replaying 68,266 requests from 393 real Claude Code sessions and 23,608 Mooncake requests failed to beat the production LRU baseline in three separate attempts, according to the study's author. The author found that under capacity pressure most recomputation comes from tool-calling loops seconds apart rather than sessions idling past a TTL, and that the TTL never fires at all. The simulator reproduced Mooncake's published hit-rate-vs-capacity table in shape but with a systematic +4–6pp offset the author could not explain across five metric definitions, which was published unresolved.", "body_md": "I replayed **68,266 requests from 393 real Claude Code sessions** and **23,608 Mooncake\nrequests** through a prefix-cache simulator, tried to beat the production baseline three\ndifferent ways, and failed. The interesting part is why: under capacity pressure, most\nrecomputation comes from tool-calling loops *seconds* apart, not from sessions idling past a\nTTL — and the TTL never fires at all.\n\nEverything here reproduces from a cold checkout with `make setup data repro`.\n\nCross-request KV prefix caching is the largest practical lever in agentic LLM serving. It's why\nyour coding agent's fiftieth turn costs a fraction of its first. Every serving stack has one —\nvLLM's automatic prefix caching, SGLang's RadixAttention, LMCache, Mooncake Store — and all of\nthem evict with LRU by default. (SGLang also ships LFU, SLRU, Priority and others behind\n`--radix-eviction-policy`; LRU is the shipped default.)\n\nThere's a large, fast-growing literature arguing LRU is the wrong policy for agentic workloads, because agent sessions go idle and LRU can't tell a paused session from a dead one. The argument is intuitive. I believed it, and built a simulator to exploit it.\n\nIt didn't work, and why it didn't work turned out to be more interesting than the policy would have been.\n\nA block-granular, discrete-event simulator of a cross-request prefix cache. Three properties that matter, and that quick implementations tend to get wrong:\n\n**Hits are prefix-contiguous.** A hit is the *longest resident prefix* of the block chain, not\na set intersection. Miss one block at depth 3 and everything after it is unusable even if it's\nstill resident.\n\n**The radix structure constrains eviction.** A block with resident children isn't evictable. So\nthe baseline is LRU *over radix leaves*, which is what SGLang and vLLM actually implement.\nBeating naive flat LRU would be a strawman.\n\n**The in-flight chain must be pinned.** See [finding 5](#5-the-harness-bug-that-makes-belady-lose-to-lru).\n\nTraces are real, not synthetic:\n\n| trace | requests | block size | hash scope | source | \n|---|---|---|---|---|\n| **SemiAnalysis AgentX** | 68,266 across **393 Claude Code sessions** | 64 tok | session-local | [HF](https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126-256k) (Apache-2.0) | \n| **Mooncake**`mooncake_trace` /`toolagent` | 23,608 | 512 tok | **global** | [GitHub](https://github.com/kvcache-ai/Mooncake) (Apache-2.0) | \n| **Mooncake**`conversation` | 12,031 | 512 tok | global | same | \n\nBefore trusting anything, I reproduced Mooncake's published hit-rate-vs-capacity table on Mooncake's own released trace, with their stated policy.\n\n| cache (blocks) | 1k | 10k | 30k | 50k | 100k | ∞ | \n|---|---|---|---|---|---|---|\n| **published (LRU)** | 0.30 | 0.40 | 0.48 | 0.50 | 0.51 | 0.51 | \n| **measured (radix-leaf LRU)** | 0.341 | 0.460 | 0.537 | 0.551 | 0.552 | 0.553 | \n| **measured (flat block LRU)** | 0.340 | 0.460 | 0.537 | 0.551 | 0.552 | 0.553 | \n\nThe shape reproduces exactly, including the saturation point they describe in prose (\"1,000 to 50,000 blocks boosts the cache hit ratio from 30% to 50%; further capacity increases show minimal improvement\").\n\n**There is a systematic +4–6pp offset I could not explain.** I tested five metric definitions —\nblock denominator, token denominator, dropping the partial tail block, per-request averaging —\nand none closes it. The infinite-cache case is policy-free, a pure property of the trace, so\nthe discrepancy is definitional or a trace-version mismatch, not a replay bug.\n\nPublishing it unresolved rather than tuning until it matches. **If you know why, please open an\nissue.**\n\n**Incidental finding:** flat block LRU and radix-leaf-restricted LRU differ by **0.02pp** on\nthis workload. The leaf restriction both major engines implement buys essentially nothing here.\n\nReproduce: `make validate`\n\n```\nsessions=393  requests=68266\n\nsession span (h):   p50=1.84  p90=28.36  max=254.8\ninter-req gap (s):  p50=2.1   p90=51.1   p99=3426.3   max=491922   (5.7 days)\n   gaps >   60s: 9.5%\n   gaps >  300s: 3.3%\n   gaps > 3600s: 1.0%\ninput tokens:       p50=88768  p90=204288  max=255808\noutput tokens:      p50=376    p90=1845\nrequests/session:   p50=70     max=3551\n\nDUTY CYCLE (fraction of wall-clock actually executing):\n   p25=3.4%   p50=13.9%   p75=33.9%\n   sessions executing <50% of lifetime: 85.5%\n```\n\nThe most-cited characterization of agentic serving reports a **20%** median duty cycle and\n**70%** of sessions below 50%. On this independent trace it's **13.9%** and **85.5%** — the\npremise is *more* extreme than published, not less.\n\nNote the shape: gaps are bimodal. A median of **2.1 seconds** (tight tool loops) with a heavy\ntail out to days.\n\nReproduce: `make characterize`\n\nThis is the finding that changed my mind.\n\nAgentSysBench ([arXiv:2608.15127](https://arxiv.org/abs/2608.15127)) reports that \"cache\nevictions contribute 55.9% of the total cache-create tokens and account for **31.5% of\naggregate monetary cost**,\" driven by a 5-minute provider TTL colliding with 1–10 minute idle\ngaps. That motivated my entire approach.\n\nSo before optimizing for it, I measured where recompute comes from — policy-independently. Replay the trace, and bucket every request's recomputed tokens by the idle gap that preceded it:\n\n| gap before request | requests | share of all recompute tokens | \n|---|---|---|\n| **<10 s** | 10,069 | **33.1%** | \n| 10–60 s | 912 | 7.0% | \n| 1–5 min | 701 | 20.5% | \n| 5–30 min | 236 | 8.6% | \n| 30–60 min | 50 | 3.0% | \n| >1 h | 123 | 5.8% | \n\n**Requests arriving after a gap longer than 5 minutes account for 17.5% of recompute.\nRequests arriving within 10 seconds account for 33.1%.**\n\nThe dominant source of cache misses here is **tight two-second tool loops whose 88k-token\nworking sets exceed cache capacity** — a *capacity* problem, not a liveness-prediction problem.\nWith a p50 gap of 2.1 seconds, almost every session is \"about to return,\" so a liveness\nestimator has essentially nothing to discriminate on.\n\nThe two numbers measure different things in different regimes, and I initially framed this as a contradiction. It isn't.\n\nAgentSysBench this repo numerator eviction-caused cache-create tokens, priced at $6.25/M recomputed prefill tokens after a >5min gap denominator **total bill** (incl. cache reads and output tokens)**all recompute tokens** regime **TTL-bound** — a provider cache where per-customer capacity is effectively unlimited and entries die on a timer**capacity-bound** — 40,000 blocks against a ~10.7M-token working set\nIn a TTL-bound cache, essentially all evictions are gap-driven by construction. My setup\nnever enters that regime — which [finding 3](#3-the-5-minute-ttl-never-fired-under-capacity-pressure)\ndemonstrates directly, since `TTL-300s` was byte-identical to LRU-leaf in every run.\n\n**Both results can be entirely correct.** The claim here is narrower and it is this: *when\ncapacity binds, it dominates the TTL, and the recompute it causes looks nothing like the\nidle-session story.* If you are provisioning cache capacity, that changes what you optimise.\nIf you are reasoning about provider TTLs, the 31.5% figure is the relevant one, not this.\n\nReproduce: `make gap`\n\n`TTL-300s` produced **byte-identical results to LRU-leaf in every single run.**\n\nLRU always evicted before the timer expired, so the TTL never became the binding constraint at any cache size I tested. This is also the cleanest evidence that these runs sit in a capacity-bound regime rather than the TTL-bound one a provider cache operates in.\n\nI implemented a policy with three separable, independently ablatable components:\n\n- **H** — hazard-based`P(session returns)` replacing recency. Online Bayesian estimator over\nobserved inter-turn gaps and continuation rates. No oracle: it only ever sees completed\nobservations.\n- **C** — physically-modelled recompute cost. Prefill cost at position*i* is a linear term\nplus an attention term proportional to*i* , so recomputing the tail of a 100k-token chain is\nfar more expensive per byte than it looks.\n- **G** — coherent session-granularity eviction. Instead of taking the*N* globally-oldest\nleaves (which may truncate 50 different chains), sacrifice one session's private tail.\n\nHit rate, 40 AgentX sessions, 4,751 requests:\n\n| cache (blocks) | LRU-leaf | TTL-300s | LFU-leaf | +H | +HC | +HCG | \n|---|---|---|---|---|---|---|\n| 8,000 | **83.48%** | 83.48% | 63.61% | 82.89% | 71.77% | 68.63% | \n| 20,000 | **93.92%** | 93.92% | 69.96% | 93.61% | 84.86% | 78.89% | \n| 50,000 | **95.76%** | 95.76% | 79.58% | 95.68% | 94.45% | 91.40% | \n\nEffective recompute cost versus LRU-leaf (negative is worse):\n\n| cache | LFU-leaf | +H | +HC | +HCG | \n|---|---|---|---|---|\n| 8,000 | −129.7% | −3.2% | −38.9% | −81.0% | \n| 20,000 | −434.3% | −4.5% | −90.4% | −207.8% | \n| 50,000 | −447.1% | −1.0% | −15.4% | −66.8% | \n\nMonotone negative. Every component made it worse, and the one I was most confident in — coherent eviction — was the worst.\n\nGiven [finding 2](#2-the-waste-isnt-where-everyone-is-looking), this is exactly what should\nhave happened. I was optimizing for a signal carrying 17.5% of the waste, using a predictor\nthat can't discriminate at a 2.1-second median gap.\n\nReproduce: `make ablation`\n\nIn my first run, Belady — an *offline oracle* — lost to LRU. That's not a result, that's a\nbroken harness, and it's worth publishing because I expect it to be common.\n\nThe cause: inserting a long chain into a near-full cache lets a policy **evict the very prefix\nit is currently building.** LRU is accidentally immune because just-inserted blocks have the\nnewest timestamp. Every non-recency policy cannibalises itself. Real engines prevent this with\nrefcount pins; a from-scratch simulator usually doesn't.\n\n**If you build one of these, make your first test \"does Belady beat LRU?\" If it doesn't, you\nhave this bug, and every policy comparison you run will be silently wrong in LRU's favour.**\n\nTwo other implementation notes:\n\n- Only the *deepest* hit block can ever be a leaf, so`touch()` need only update that one\nblock. An O(chain length) walk becomes O(1) — which matters at AgentX's 1,387-block median.\n- Score eviction candidates by sampling *k* least-recently-used leaves rather than scanning the\ncache. This is what production caches do anyway, so it's realism, not a shortcut.\n\n**Which constraint binds determines what you should optimise, and the two regimes want\nopposite things.** If your cache is TTL-bound, liveness prediction and retention policy are the\nlevers, and the published eviction-cost work applies directly. If it's capacity-bound — which\nis where these runs sit — the question isn't \"will this session come back?\" but \"how do I fit\n88k-token working sets for N concurrent sessions in tight tool loops?\" That points at\ncompression, tiering, admission control and working-set-aware scheduling instead, and liveness\nprediction has essentially nothing to work with at a 2.1s median gap.\n\nI went in assuming the liveness framing and it cost me three failed policies. Establishing which regime you're in first would have saved all of it.\n\n**LRU-leaf is a stronger baseline than the literature treats it as.** I couldn't beat it with\nthree independent mechanisms on real traces. Meanwhile several published alternatives are\nevaluated against degraded ports of their competitors — two separate papers benchmark against\nContinuum with its adaptive TTL replaced by a fixed 2s or 0.3s pin, which disables the thing\nthat makes it work. This null result suggests those margins are softer than they read.\n\n**Validate against a published curve before trusting your own numbers.** Doing that surfaced a\ndiscrepancy I still can't explain, and it's the only reason I trust anything else here.\n\n- **These runs are capacity-bound, not TTL-bound.** 40,000 blocks against a ~10.7M-token\nworking set. A provider cache like Anthropic's is the opposite: per-customer capacity is\neffectively unlimited and entries die on a 5-minute timer. Findings 2 and 3 characterise the\ncapacity-bound regime and say nothing about the TTL-bound one.\n- **This is simulation.** It models cache policy faithfully and GPU execution not at all. Valid\nfor \"what should I keep in cache\";**not** valid for throughput, latency, or SLO attainment.\n- **AgentX block hashes are session-local** , so they're namespaced per session. That models*zero* cross-session sharing — conservative, but it means shared system prompts across users\nare invisible here. Mooncake's hashes are global but its trace is one dense hour with no idle\nstructure.\n- **AgentX session arrival times are synthesised** (uniform over a window), because the trace\nstores session-relative timestamps only.\n- **393 sessions and one hour of Mooncake is not the world.**\n- **I am not claiming the liveness literature is wrong.** I'm claiming that in a\ncapacity-bound cache the lever it targets has little to work with, and that establishing\nwhich regime you're in should come before choosing a policy.\n\n```\ngit clone https://github.com/<you>/agentic-kv-cache && cd agentic-kv-cache\nmake setup      # venv\nmake data       # ~1.1 GB of traces (Apache-2.0), then flattens AgentX to a pickle\nmake repro      # all four experiments, writes results/\n```\n\nIndividually:\n\n``` php\nmake validate      # Mooncake reproduction        -> results/01_validate.txt\nmake characterize  # AgentX duty cycle and gaps   -> results/02_characterize.txt\nmake gap           # recompute by idle gap        -> results/03_gap.txt\nmake ablation      # policy ablation              -> results/04_ablation.txt\n```\n\nThe simulator is pure stdlib Python; `numpy` is only used by helper scripts. Committed outputs\nin [`results/`](https://github.com/gauravapiscean/agentic-kv-cache/blob/main/results) let you check the tables without downloading anything.\n\nIf you can answer any of these, please open an issue — I'd genuinely like to know:\n\n1. **Why the +4–6pp Mooncake offset?** Policy-free at infinite cache, so it should be\nexplicable by metric definition alone, and five definitions don't close it.\n2. **Is there a workload where liveness-aware eviction beats radix-leaf LRU?** Plausibly one\nwith much longer median gaps than 2.1s — human-in-the-loop approval flows, perhaps.\n3. **Does the 33%-from-sub-10-second-gaps result hold on other agentic traces?** If it does,\na good chunk of this subfield is aimed at the wrong term.\n\nTraces: [Mooncake](https://github.com/kvcache-ai/Mooncake) (Moonshot AI, FAST'25) and the\n[AgentX corpus](https://huggingface.co/datasets/semianalysisai) (SemiAnalysis), both Apache-2.0.\nThis work is independent of and unaffiliated with either.\n\nMIT licensed.", "url": "https://wpnews.pro/news/lru-is-harder-to-beat-than-the-kv-cache-papers-suggest", "canonical_source": "https://github.com/gauravapiscean/agentic-kv-cache", "published_at": "2026-09-10 13:39:11+00:00", "updated_at": "2026-09-12 13:40:33.551700+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-agents", "mlops"], "entities": ["Claude Code", "Mooncake", "vLLM", "SGLang", "LMCache", "SemiAnalysis AgentX"], "alternates": {"html": "https://wpnews.pro/news/lru-is-harder-to-beat-than-the-kv-cache-papers-suggest", "markdown": "https://wpnews.pro/news/lru-is-harder-to-beat-than-the-kv-cache-papers-suggest.md", "text": "https://wpnews.pro/news/lru-is-harder-to-beat-than-the-kv-cache-papers-suggest.txt", "jsonld": "https://wpnews.pro/news/lru-is-harder-to-beat-than-the-kv-cache-papers-suggest.jsonld"}}