cd /news/ai-infrastructure/aws-distributed-kv-caching-skill-cut… · home › topics › ai-infrastructure › article
[ARTICLE · art-173595] src=explainx.ai ↗ pub= topic=ai-infrastructure verified=true sentiment=↑ positive

AWS Distributed KV Caching Skill: Cut Self-Hosted LLM Prefill Cost With ElastiCache

Amazon Web Services on November 20, 2026 published an Agent Toolkit skill that lets coding agents evaluate and deploy distributed KV caching for self-hosted vLLM and SGLang fleets, using Amazon ElastiCache for Valkey as a shared L2 tier beside the GPU servers. AWS cites benchmarks of up to 40% inference cost reduction and 80% faster time to first token when prompts repeat above a length cutover, and gives a sizing example of two cache.i8ge.48xlarge nodes for eight p5 hosts on 64K legal prompts. The skill runs a read-only fit check, a KVTrace ROI replay, LMCache connector patching for vLLM/SGLang including llm-d and Ray Serve workers, and requires explicit human approval for create, change and delete operations before production promotion.

key takeaway

Add an LMCache connector to existing vLLM or SGLang workers so any node can RDMA-load shared KV blocks from ElastiCache, but force one KV ID hash mechanism and explicit KV dtype across servers to avoid key mismatch that shows healthy hit counters with wrong answers.

by read8 min views4 publishedNov 21, 2026
AWS Distributed KV Caching Skill: Cut Self-Hosted LLM Prefill Cost With ElastiCache
Image: Explainx (auto-discovered)

Amazon Web Services on November 20, 2026 published an Agent Toolkit skill that lets coding agents evaluate and deploy distributed KV caching for self-hosted vLLM and SGLang fleets. The problem is familiar: each GPU server keeps an isolated prefix cache, so a user who resumes a long agent thread on another node pays full prefill again. AWS says Amazon ElastiCache for Valkey can hold a shared L2 tier beside the GPU fleet, with cited benchmarks up to 40% inference cost reduction and 80% faster time to first token (TTFT) when prompts repeat above a length cutover.

If you run vLLM vs SGLang comparisons on p5 instances, this skill is AWS’s guided path from diagnosis to LMCache wiring—not a new inference engine, but agent-operated infrastructure setup.

TL;DR #

Step What the skill does (AWS blog, Nov 20, 2026)
Fit check Read-only EC2 describe; count EFA ; gather model, TP size, KV dtype, routing
ROI gate Run KVTrace on your traffic; stop if savings do not cover cache nodes
Sizing Example: two cache.i8ge.48xlarge nodes for eight p5 hosts on 64K legal prompts
Engine hook Add LMCache connector to existingvLLM/SGLang , including llm-d and Ray Serve workers
Consistency Force one KV ID hash mechanism and explicit KV dtype across servers
Validation Empty local tiers; verify two cache-loaded runs match before production
Agents Works with Claude Code ,Codex ,Kiro , other Agent Skills hosts

Why isolated GPU caches waste money #

AWS opens with a legal assistant example: hundreds of thousands of tokens of case history per conversation. When the attorney returns after lunch, the GPU that picks up the session has no KV state for that transcript. The model recomputes attention keys and values for every prior token—prefill—before generating a single new word. The same pattern hits coding agents, document Q&A, and any multi-turn agent harness that resends full context each turn.

Local prefix caching on one server only helps when session affinity keeps the user on that host. Routing shifts, autoscaling, and failures break affinity. Distributed KV caching stores reusable KV blocks in ElastiCache so any inference worker can RDMA-load them.

What the Agent Toolkit skill automates #

AWS built the skill so a coding agent can run the workflow in conversational steps:

Fit check before spend — describe-instances, EFA count, and questions the API cannot answer (engine, tensor parallel size, KV dtype, server price, prompt repeat rate). The agent muststop when constraints fail. 2. KVTrace ROI — offline replay of cache events without deploying cache hardware. AWS stresses traces containhashes and counts , not prompts, with optional--anonymize-key-file . 3. LMCache integration — patch existing vLLM or SGLang stacks, including workers underllm-d andRay Serve , without replacing routing logic. 4. Guardrails — matching KV ID seeds and dtypes to preventkey mismatch (healthy hit counters with wrong answers). Production promotion requires explicit human approval for create/change/delete operations.

The skill ships from the Agent Toolkit for AWS with the AWS MCP Server—the same pattern as other agent skills teams mount in Claude Code.

Flow 1: Diagnosing slow first token #

AWS’s sample prompt: a vLLM fleet serving 64K-token legal contexts on p5 instances with 13-second TTFT because the GPU recomputes history every turn.

The agent identifies cold prefill, checks whether enabling prefix caching alone fixes back-to-back turns, and flags tenant isolation: shared caches can leak timing signals between tenants. AWS recommends separate caches or tenant-specific KV hash salts when cases must stay isolated.

AWS distributed KV caching agent skill diagram for legal assistant GPU fleets

Image: Amazon Web Services

Flow 2: KVTrace savings estimates #

For eight p5 instances and 64,000-token RAG prompts, the blog’s agent looks up GPU and cache node prices, derives prefill throughput and KV bytes from benchmarks, and proposes two cache.i8ge.48xlarge nodes. When the operator omits a cache hit rate, the agent shows scenarios and recommends a KVTrace run. KVTrace replays recorded events against candidate cache sizes and recommends the smallest capacity that reaches 90% of the savings ceiling in AWS’s synthetic demo report.

KVTrace report chart for distributed KV caching cost savings on self-hosted LLMs

Image: Amazon Web Services AWS also notes freed GPU cycles only become dollars if you scale down servers or serve more traffic on the same fleet—an easy mistake in capacity planning.

Flow 3: Standing up ElastiCache Valkey #

On confirmation, the agent creates security groups with EFA permissions, a Valkey 9.1 cluster with eviction-tuned parameter groups, LMCache cache users with Secrets Manager passwords, and per-GPU LMCache servers. Health checks use small test objects and repeated prompts to confirm RDMA loads from ElastiCache.

Detailed commands live in Amazon ElastiCache distributed KV caching documentation.

Flow 4: Troubleshooting RDMA errors #

AWS’s troubleshooting prompt: LMCache connects and BLOB.RDMA_HELLO succeeds, but INFO largeobj shows climbing rdma_write_errors. The skill points to security groups that block EFA traffic between members, with CLI steps to inspect and fix rules—exactly the class of failure that shows up when agents automate infra without network diagrams.

Prerequisites checklist #

Requirement Version or SKU (per AWS post)
Coding agent Agent Skills support (Claude Code, Codex, Kiro, etc.)
GPUs p5 or otherEFA RDMA write instances
LMCache 0.5.8rc1+
vLLM 0.11.1+
SGLang 0.5.13+
ElastiCache Valkey 9.1+ oncache.i8ge.48xlarge

How this fits the inference stack landscape #

Distributed KV caching is complementary to:

Teams on Together serverless pricing or hyperscaler APIs may skip self-hosting entirely; this skill targets operators who already pay for p5 capacity and need agent threads to stop recomputing million-token contexts.

Cutover math: when cache beats recompute #

AWS’s blog stresses a cutover point in prompt length: below it, reading KV blocks from ElastiCache can take longer than recomputing attention on-GPU. Above it—and when the same prefixes repeat across sessions—shared L2 wins. Legal, coding, and support bots with static system prompts plus growing user threads usually sit above that threshold faster than single-shot Q&A.

The skill forces the agent to cite where each number came from (price list lookups, benchmark-derived prefill rates, KVTrace scenarios). That matters because vendors sometimes market cache products with fleet-wide averages that hide low-repeat traffic segments. Run KVTrace on a full business day per AWS’s guidance before approving cache.i8ge nodes.

Operational checklist before production traffic #

  1. Harmonize dtypes and hash seeds across every vLLM worker—including Ray or llm-d sidecars.
  2. Drain local tiers and run duplicate prompts; mismatched tensor layouts can pass health checks yet change outputs.
  3. Measure L2 hit share against KVTrace prediction after the cluster is live.
  4. Document tenant isolation decisions if multiple customers share one GPU fleet.
  5. Keep prefix caching enabled where session affinity is high; distributed cache complements rather than replaces on-box caches.

Security and tenancy notes #

AWS explicitly warns that shared KV caches can let one tenant infer another’s repeated text from timing unless you shard caches or salt hashes. For regulated workloads, treat cache tiers like PCI-scoped Redis: separate clusters per tenant class, audit RDMA security groups, and rotate Secrets Manager passwords LMCache users consume.

Agent skills that create cloud resources should stay behind human approval gates—the skill’s design asks the agent to request confirmation before creating billable ElastiCache nodes.

How to start the skill in your agent #

AWS tells operators to pull the skill from the Agent Toolkit for AWS repository or install the AWS MCP Server, then describe the latency problem in natural language (for example the 13-second legal-assistant TTFT example). The agent should load the skill only after approval in environments like Kiro.

Pair the skill with your existing observability: track TTFT p50/p95 and GPU prefill seconds before and after cache enablement. Without those baselines, finance teams cannot verify AWS’s 40% cost headline on your fleet.

If you already run SageMaker HyperPod across regions, deploy cache clusters per GPU pool—cross-region KV reuse is not described in the November 20 post.

When to skip ElastiCache entirely #

Stay on single-node prefix caching when:

  • Sessions stick to one GPU with greater than 90% prefix hit rate already.
  • Median prompts stay below the KVTrace cutover for your model width.
  • Traffic is mostly unique one-shot queries with no repeated system prompts.
  • Regulatory policy forbids shared cache tiers between customers.

The skill is designed to stop early in those cases rather than provision cache.i8ge hardware by default.

Sources #

Primary: Cut self-hosted LLM cost and latency with the distributed KV caching agent skill (AWS ML Blog, November 20, 2026). Technical companion: Announcing distributed KV caching for self-hosted LLMs with Amazon ElastiCache.

Specifications and versions are accurate as of November 21, 2026.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @amazon web services 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/aws-distributed-kv-c…] indexed:0 read:8min 2026-11-21 · —