Harnessing LMCache: Building a Pluggable KV-Cache Management & Testing Framework

A developer built a test and implementation framework around LMCache, an open-source KV-cache management layer for LLM inference that stores and reuses attention key-value tensors outside the serving engine. LMCache chunks token streams into 256-token blocks hashed with blake3 for prefix reuse and tiers cache snapshots across GPU/CPU memory, local disk, and remote backends such as Redis and S3, with support for vLLM, SGLang, and TensorRT-LLM. The project aims to cut prefill recomputation and time-to-first-token for long-context RAG and agentic workloads.

LMCache test & implementation. Introduction As Large Language Model LLM architectures scale and context windows expand to tens of thousands of tokens, prefill latency has become a major bottleneck in AI application performance. Processing long, repeated prompt prefixes—such as system prompts, core reference documents in Retrieval-Augmented Generation RAG , and extensive agent chat histories—consumes substantial compute cycles and drives up time-to-first-token TTFT . What is LMCache? LMCache https://github.com/LMCache/LMCache is an open-source Key-Value KV cache management layer designed to solve this exact bottleneck. Rather than recomputing attention Key and Value tensors for every incoming request, LMCache captures, indexes, and stores these tensors outside the core inference engine, re-injecting them on demand. Key features of LMCache include: - Chunk-Level Prefix Reuse: Token streams are partitioned into fixed-size chunks defaulting to 256 tokens . Each chunk is assigned a rolling prefix hash using blake3 , guaranteeing that matching prefixes immediately hit cached states. - Tiered Storage Architecture: Cache snapshots flow seamlessly across device memory GPU/CPU RAM , local disk L2 , and remote backends e.g., Redis, S3, Mooncake, NIXL with Least Recently Used LRU eviction policies. - Engine Independence: Cache snapshots survive engine restarts and slot evictions, allowing cross-instance sharing and disaggregated prefill/decode topologies. The LMCache Ecosystem The LMCache ecosystem spans mainstream open-source projects, enterprise hardware vendors, and infrastructure orchestrators: - Inference Engines: Native support targets server-grade Linux/CUDA engines such as vLLM, SGLang, and TensorRT-LLM. - Orchestration Frameworks: Integrates with orchestrators like NVIDIA Dynamo, vLLM Production Stack, llm-d , and KServe. - Hardware & Storage Ecosystem: Compatible with storage vendors Dell, Redis, DDN, Valkey, WEKA, VAST, Samsung, IBM Storage and accelerator providers NVIDIA, AMD, Ascend, Intel, Moore Threads, Rebellions, MetaX . - Research Extensions: Incorporates advanced research ideas including TurboQuant, CacheBlend, CacheGen, and Cartridges. Excerpt from official Github repository of the project: LMCache is a KV cache management layer for LLM inference. It turns KV cache from a temporary state into reusable AI-native knowledge that can be stored persistently, reused across multiple serving engines, monitored with an observability stack, and transformed for better generation quality. As a result, LMCache reduces TTFT time-to-first-token and improves throughput , especially for long-context agentic, multi-turn conversation, and knowledge-augmented workloads e.g., RAG . LMCache is vendor-neutral . It can be used as a KV cache layer for a range of mainstream open-source serving engines, inference frameworks, hardware vendors, storage systems, and infrastructure providers. The vendor neutrality allows users to freely switch between serving engines and storage vendors, while reusing the stored KV caches. Key features - Engine-independent deployment : LMCache, as a standalone daemon process, manages KV cache independently from the inference engine process, so that KV cache will not be lost even if the inference engine crashes i.e., no fate-sharing with engines . - Persistent, tiered KV cache offloading and reuse : Move KV caches out of GPU memory into a tiered storage hierarchy spanning CPU memory, local storage, and remote backends, enabling reuse across requests, sessions, and engine instances to reduce repeated prefill computation and improve TTFT. - Production-level KV cache observability : LMCache provides a rich set of KV cache observability metrics, including typical Kubernetes metrics health monitoring, performance diagnostics , KV-cache-specific metrics request-level and token-level prefix cache hits, lifecycle, request-level KV cache performance , management metrics user-specific usage , and more. - Pluggable storage and transport backends : Easily integrate remote storage and KV transfer backends through a unified interface, enabling KV cache offloading and sharing across storage providers. Through this interface, LMCache supports storage backends including CPU RAM, local disk SSD , Redis/Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL, and GDS. - Non-prefix KV reuse : Extend KV reuse beyond prefix caching by reusing cached KV blocks at any position in the prompt. This leverages CacheBlend to selectively recompute tokens for quality recovery. - PD disaggregation and KV transfer : Support KV cache transfer from prefill workers to decode workers over NVLink, RDMA, or TCP through transport layers such as NIXL. - Pluggable KV transformation : A simple interface for researchers to write compression, token dropping, and custom serialization through a flexible SERDE interface. LMCache is becoming an integral layer in the LLM inference ecosystem , with community -driven integration with serving engines, inference frameworks, hardware vendors, storage systems, and infrastructure providers: Implementation Disclaimer: Officially, LMCache targets server-side GPU inference engines vLLM, SGLang, TRT-LLM on Linux; official PyPI wheels require CUDA, and feature requests for Ollama/LM Studio support e.g., issue 923 were closed as "not planned". However, for a mock test and rapid local prototyping, an LMCache-style framework named KVBridge was implemented to bring these concepts to my local test environment. Application overview & purpose KVBridge demonstrates the core value proposition of LMCache https://github.com/LMCache/LMCache — a KV cache management layer for scalable LLM inference — on a developer laptop. LMCache itself targets server-side engines vLLM, SGLang, TensorRT-LLM and needs CUDA. KVBridge re-implements the LMCache concepts chunked, content-addressed KV reuse; tiered storage; engine-independent keys on top of the engines developers actually run: two local ones llama.cpp, Ollama and any OpenAI-compatible cloud service, all behind one pluggable engine registry. What KVBridge is, and is not KVBridge is an LMCache- style implementation. It is not the LMCache package. It optionally reuses LMCache's native TokenHasher when the real package can be built scripts/install lmcache.sh ; otherwise a bit-compatible blake3 rolling hash is used. System Architecture & Components KVBridge mirrors LMCache's core mechanisms rolling prefix hashing, tiered storage, and cache persistence while adding an extensible engine registry to support local engines llama.cpp , Ollama and any OpenAI-compatible cloud provider. Code Layout The local application, uses Gradio for the UI implementation and is configurable to connect to multiple backends. lmcache-test/ ├── kvbridge/ │ ├── init .py │ ├── main .py python -m kvbridge │ ├── app.py Gradio GUI registry-driven , AppState, main , SIGTERM handler │ ├── backends.py engine registry, LlamaCppBackend, OllamaBackend, CloudBackend, make backend │ ├── engine.py KVBridgeEngine: lookup, restore, compute, store │ ├── store.py L1 RAM / L2 disk tiered store │ ├── hashing.py chunking + chained hashes │ ├── metrics.py counters, TTFT, hit rates │ ├── charts.py Plotly figures │ ├── scenarios.py demo scenario registry │ └── config.py Settings GUI/env/.env , presets, update env file ├── tests/ unit + live tests ├── scripts/ setup, start, stop, test, cleanup, install lmcache ├── Docs/ Quickstart, Architecture, Research, this file ├── input/ output/ tracked via .gitkeep, contents ignored ├── requirements.txt ├── pytest.ini ├── .env.example every KVB variable, local and cloud examples └── README.md includes the Apache-2.0 licence statement Evaluation Modes To evaluate performance gains, the architecture routes requests through three distinct operational modes: 1. cold : Disables native prompt caching cache prompt=false and wipes existing slot state, forcing a full prefill calculation on every request to establish a baseline. 2. native : Relies strictly on the engine's built-in, in-slot prefix caching cache prompt=true . 3. lmcache : Drops in-slot state, performs chunk-hash lookups against the L1/L2 store, restores cached tensor state into the engine slot, pre-fills only the un-cached tail suffix, and saves new chunk snapshots upon completion. - Full KV State Export and Import engine.py : the primary pipeline manages KV tensor persistence using only for mock test purpose llama.cpp 's /slots/{id}?action=save|restore API: ""The cache-aware inference pipeline. Request flow llama.cpp, full integration :: prompt --tokenize-- tokens --chunk+hash-- h0 h1 h2 ... | store.lookup longest prefix <-+ |hit |miss fetch snapshot L1 RAM / L2 disk | restore into engine slot | +---------+----------+ v engine.generate prefills only the missing suffix v save slot - store.put only when new chunks were computed Three execution modes make the benefit measurable: cold prefix reuse disabled - every request prefills the whole prompt. native the engine's own slot/prefix cache only baseline a user has today . lmcache native cache is dropped first modelling eviction by another session, a restart, ... and the KV layer restores state instead. """ ... def infer full self, prompt: str, rec: RequestRecord, n predict: int, temperature: float - None: """llama.cpp path: real KV tensor save/restore handling.""" b = self.backend tokens = b.tokenize prompt if rec.mode == "cold": b.reset native cache self. fill rec, b.generate prompt, tokens, n predict, False, temperature return if rec.mode == "native": self. fill rec, b.generate prompt, tokens, n predict, True, temperature return ---- LMCache Mode Pipeline hashes = self.hasher.chunk hashes tokens rec.chunks total = len hashes rec.chunk status = self.store.chunk status hashes Drop native slot state to isolate LMCache retrieval benefit b.reset native cache match = self.store.lookup hashes rec.lmcache match tokens, rec.chunks hit, rec.tier = match.tokens, match.chunks, match.tier Restore matching KV snapshot from L1 RAM or L2 Disk if match.entry is not None: try: blob = self.store.fetch match.entry n = b.restore state blob rec.restore ms = time.perf counter - t0 1000 except BackendError, KeyError, OSError as exc: log.warning "Restore failed %s ; falling back to recompute", exc rec.chunks hit, rec.lmcache match tokens, rec.tier = 0, 0, "none" b.reset native cache Compute generation prefills missing suffix only res = b.generate prompt, tokens, n predict, True, temperature self. fill rec, res Persist newly computed tail chunks into the Tiered Store if len hashes rec.chunks hit: blob, n saved = b.save state self.store.put hashes, len hashes self.s.chunk size, blob, note=f"{n saved} tok snapshot" - Extensible Engine Registry backends.py : to prevent hardcoded conditionals across engine implementations, a centralized registry handles dynamic initialization: """Inference-engine adapters. Three adapters share one small interface :class: Backend and are looked up through a registry :func: register engine , :func: resolve engine , so the engine is chosen purely by configuration KVB ENGINE or the GUI : LlamaCppBackend Talks to llama-server llama.cpp . llama.cpp exposes the slot save/restore API POST /slots/{id}?action=save|restore|erase which serialises the real KV tensors of a sequence. That is the integration point that lets an external cache layer LMCache semantics take KV out of the engine and inject it back - full integration . OllamaBackend Talks to the Ollama REST API. Ollama deliberately hides KV state: there is no KV export/import and no tokenizer endpoint. The adapter therefore works in shadow mode - the LMCache layer indexes/accounts for chunks and predicts reuse, while Ollama's own prefix cache does the real work and its prompt eval cached count is used as ground truth. CloudBackend Talks to any OpenAI-compatible HTTP API OpenAI, Groq, OpenRouter, Together, Mistral, DeepSeek, vLLM, LM Studio, ... . Hosted engines never expose their KV tensors either, so it also runs in shadow mode ; the provider's own prompt-cache counter cached tokens is the ground truth. Prompts leave the machine - the GUI says so explicitly. """ from future import annotations import json import logging import os import re import shutil import signal import subprocess import time import uuid import zlib from abc import ABC, abstractmethod from dataclasses import dataclass from pathlib import Path from typing import Optional, Sequence from urllib.parse import urlparse import requests from .config import CLOUD API STYLES, DEFAULT LLAMA SEARCH PATHS, Settings ... @dataclass frozen=True class EngineSpec: key: str Identifier e.g., KVB ENGINE label: str GUI Label cls: type Backend Adapter implementation class blurb: str = "" aliases: tuple str, ... = REGISTRY: dict str, EngineSpec = {} def register engine spec: EngineSpec - EngineSpec: REGISTRY norm spec.key = spec return spec def make backend kind: str, settings: Settings - Backend: return resolve engine kind .cls settings Engine Registration register engine EngineSpec "llamacpp", "llama.cpp local ", LlamaCppBackend, "Full KV save/restore through llama-server slots.", "llama", register engine EngineSpec "ollama", "Ollama local ", OllamaBackend, "Shadow mode: Ollama hides its KV cache.", register engine EngineSpec "cloud", "Cloud / OpenAI-compatible API", CloudBackend, "Shadow mode against hosted engines.", "openai", "api" - Chained Rolling Prefix Hasher hashing.py : token streams are packed into big-endian uint32 representations and processed sequentially: """Chunk-level rolling prefix hashing, compatible with LMCache. LMCache splits a token sequence into fixed-size chunks default 256 tokens and identifies each chunk by a rolling prefix hash :: h 0 = blake3 prefix seed || pack chunk 0 h i = blake3 h {i-1} || pack chunk i where pack is the big-endian uint32 encoding of the token ids lmcache/v1/multiprocess/token codec.py . Only complete chunks are hashed; a trailing partial chunk is discarded - exactly like lmcache.v1.multiprocess.token hasher.TokenHasher . This module re-implements that algorithm so the app runs on macOS where the full LMCache wheel cannot be installed and can optionally delegate to the real TokenHasher when the lmcache package is importable. A unit test verifies both produce identical digests. """ from future import annotations import hashlib import logging import struct from typing import Protocol, Sequence log = logging.getLogger name try: blake3 is the algorithm LMCache uses import blake3 as blake3 def digest data parts: Sequence bytes - bytes: h = blake3.blake3 for part in data parts: h.update part return h.digest ALGORITHM = "blake3" except ImportError: pragma: no cover - fallback keeps the app usable def digest data parts: Sequence bytes - bytes: h = hashlib.sha256 for part in data parts: h.update part return h.digest ALGORITHM = "sha256 blake3 missing; NOT LMCache compatible " def pack tokens tokens: Sequence int - bytes: """Pack token ids as big-endian uint32 LMCache wire format .""" return struct.pack f" {len tokens }I", tokens def namespace seed namespace: str - bytes: """Derive the initial prefix hash from a model/config namespace. KV tensors are only valid for the exact model, quantisation and context configuration that produced them, so the namespace is mixed into the root of every hash chain LMCache keys also embed the model name . """ return digest namespace.encode "utf-8" class Hasher Protocol : chunk size: int impl: str def chunk hashes self, tokens: Sequence int - list bytes : ... class ChunkHasher: """Pure-Python implementation of LMCache's rolling chunk hashing.""" impl = "kvbridge LMCache-compatible blake3 " def init self, chunk size: int = 256, namespace: str = "" - None: if chunk size < 1: raise ValueError "chunk size must be = 1" self.chunk size = chunk size self.namespace = namespace self.seed = namespace seed namespace def chunk hashes self, tokens: Sequence int - list bytes : """Return one rolling hash per complete chunk of tokens .""" n complete = len tokens - len tokens % self.chunk size prefix = self.seed out: list bytes = for i in range 0, n complete, self.chunk size : prefix = digest prefix, pack tokens tokens i : i + self.chunk size out.append prefix return out class LMCacheHasherAdapter: """Delegates to lmcache.v1.multiprocess.token hasher.TokenHasher .""" def init self, chunk size: int = 256, namespace: str = "" - None: from lmcache.v1.multiprocess.token hasher import TokenHasher type: ignore self.chunk size = chunk size self.namespace = namespace self.seed = namespace seed namespace self. hasher = TokenHasher chunk size=chunk size, hash algorithm="blake3" self.impl = "lmcache.TokenHasher real LMCache " def chunk hashes self, tokens: Sequence int - list bytes : return self. hasher.compute chunk hashes list tokens , prefix hash=self.seed def make hasher chunk size: int, namespace: str, prefer lmcache: bool = False - Hasher: """Create a hasher, preferring the real LMCache one when requested.""" if prefer lmcache: try: return LMCacheHasherAdapter chunk size, namespace except Exception as exc: ImportError or LMCache init problems log.warning "LMCache TokenHasher unavailable %s ; using built-in hasher", exc return ChunkHasher chunk size, namespace def common prefix len a: Sequence int , b: Sequence int - int: """Length of the longest common prefix of two token sequences.""" n = min len a , len b i = 0 while i < n and a i == b i : i += 1 return i Configuration examples & usage scenarios Pick one block for .env or enter the same values in the GUI and press Connect . - .env — local, full integration llama.cpp : KVB ENGINE=llamacpp KVB OLLAMA MODEL=llama3.2:latest used to locate the GGUF blob KVB CTX SIZE=8192 KVB LLAMA START TIMEOUT=300 slow disk? allow more time - .env — local, shadow mode Ollama : KVB ENGINE=ollama KVB OLLAMA URL=http://127.0.0.1:11434 KVB OLLAMA MODEL=llama3.2:latest KVB OLLAMA KEEP ALIVE=30m - .env — hosted provider key stays in your shell : KVB ENGINE=cloud KVB CLOUD PROVIDER=openai KVB CLOUD MODEL=gpt-4o-mini KVB CLOUD API KEY ENV=OPENAI API KEY export OPENAI API KEY=... before start.sh KVB CLOUD TOKEN BUDGET=200000 spend guard - .env — self-hosted vLLM / any OpenAI-compatible server: KVB ENGINE=cloud KVB CLOUD BASE URL=https://llm.example.internal/v1 KVB CLOUD MODEL=meta-llama/Llama-3.1-8B-Instruct KVB CLOUD API KEY ENV=INTERNAL LLM KEY KVB CLOUD EXTRA HEADERS={"X-Team": "search"} - .env — free smoke test of the cloud path no account : KVB ENGINE=cloud KVB CLOUD PROVIDER=ollama-local-openai http://127.0.0.1:11434/v1 KVB CLOUD MODEL=llama3.2:latest | Scenario | Engine | What to do | What you see | |---|---|---|---| | A. Maximum benefit, fully offline | llama.cpp | KVB ENGINE=llamacpp , Connect, run Document Q&A | First question prefills seconds ; later ones restore KV in tens of ms; Engine restart shows an L2 restore | | B. Observe an existing Ollama setup | Ollama | KVB ENGINE=ollama , Connect, run Shared system prompt | Predicted reuse next to prompt eval cached count ; SHADOW label | | C. Measure a hosted provider's prefix cache | Cloud | Preset + model + key + token budget, Connect, run Shared system prompt with a long input document | cached tokens reported by the provider per request; cold adds a nonce and shows no reuse | | D. Try the cloud path for free | Cloud | Preset ollama-local-openai , model llama3.2:latest | Same GUI and metrics, no key, no cost | | E. Switch engines without editing files | any | Change the engine radio, edit fields, Connect, optionally Save settings to .env | Status line states engine, location and FULL/SHADOW mode | - Programmatic equivalent: python from kvbridge.config import Settings from kvbridge.backends import make backend from kvbridge.engine import CachedEngine s = Settings.from env honours KVB ENGINE and .env assert not problems := s.validate , problems engine = CachedEngine s, make backend s.engine, s engine.start Final Disclaimer and findings after the tests 🚩 As noted earlier, this application was designed to demystify LMCache functionality and practical usage. Operating as a local, GPU-free implementation, it offers a purely educational pathway to observe firsthand how tiered caching and prefix reuse enhance inference efficiency. | Question | Answer | |---|---| | Does LMCache officially support Ollama / llama.cpp? | No Supported: vLLM, SGLang, TRT-LLM. Issue 923 Ollama / LM Studio was closed “not planned”. | | Can the PyPI wheel run on macOS? | No Wheels are manylinux-only and require CUDA CUDA HOME not set . | | Can LMCache's hasher be used standalone? | Partly With NO GPU EXT=1 plus extra dependencies the native TokenHasher builds. | | Can llama.cpp export/import KV state? | Yes /slots/{id}?action=save | | Can a hosted cloud API export/import KV state? | No Never. Providers run their own prefix cache and report {% raw %} cached tokens , enabling the same accounting-only shadow mode . | | Can Ollama export/import KV state? | No No public KV API. It does report prompt eval cached count , enabling an accounting-only shadow mode . | Conclusion Building a practical testbed around LMCache concepts provides valuable insights into modern prompt caching mechanics. Key takeaways from the project include: 1. Measurable TTFT Reduction : In benchmark scenarios e.g., ~1.9k-token document Q&A , average TTFT dropped from 4,554 ms cold to 999 ms lmcache —achieving a 4.6x overall speedup across a 12-request workload. On warm hits, TTFT fell as low as 11 ms L1 RAM and 13 ms L2 Disk . 2. Persistence vs. Native In-Slot Caching : While native engine caching can be faster on a single warm slot e.g., 64 ms , the primary value of an LMCache-style architecture is capacity and persistence—snapshots survive process restarts, span arbitrary numbers of prefixes, and remain usable across multiple sessions. 3. Observability & Control : Through an intuitive Gradio GUI, automated test suites over 180 unit/live tests , and robust SIGTERM lifecycle handling, developers can reliably test, visualize, and optimize prompt reuse strategies on any workstation. Thanks for reading 🌒 Links - Code repository for this post: https://github.com/aairom/lmcache-test https://github.com/aairom/lmcache-test - LMCache repository: https://github.com/LMCache/LMCache https://github.com/LMCache/LMCache