Harnessing LMCache: Building a Pluggable KV-Cache Management & Testing Framework

wpnews.pro

LMCache test & implementation.

#

Introduction

As Large Language Model (LLM) architectures scale and context windows expand to tens of thousands of tokens, prefill latency has become a major bottleneck in AI application performance. Processing long, repeated prompt prefixes—such as system prompts, core reference documents in Retrieval-Augmented Generation (RAG), and extensive agent chat histories—consumes substantial compute cycles and drives up time-to-first-token (TTFT).

#

What is LMCache?

LMCache is an open-source Key-Value (KV) cache management layer designed to solve this exact bottleneck. Rather than recomputing attention Key and Value tensors for every incoming request, LMCache captures, indexes, and stores these tensors outside the core inference engine, re-injecting them on demand.

Key features of LMCache include:

Chunk-Level Prefix Reuse: Token streams are partitioned into fixed-size chunks (defaulting to 256 tokens). Each chunk is assigned a rolling prefix hash (using blake3 ), guaranteeing that matching prefixes immediately hit cached states.
Tiered Storage Architecture: Cache snapshots flow seamlessly across device memory (GPU/CPU RAM), local disk (L2), and remote backends (e.g., Redis, S3, Mooncake, NIXL) with Least Recently Used (LRU) eviction policies.
Engine Independence: Cache snapshots survive engine restarts and slot evictions, allowing cross-instance sharing and disaggregated prefill/decode topologies.

#

The LMCache Ecosystem

The LMCache ecosystem spans mainstream open-source projects, enterprise hardware vendors, and infrastructure orchestrators:

Inference Engines: Native support targets server-grade Linux/CUDA engines such as vLLM, SGLang, and TensorRT-LLM.
Orchestration Frameworks: Integrates with orchestrators like NVIDIA Dynamo, vLLM Production Stack, llm-d , and KServe.
Hardware & Storage Ecosystem: Compatible with storage vendors (Dell, Redis, DDN, Valkey, WEKA, VAST, Samsung, IBM Storage) and accelerator providers (NVIDIA, AMD, Ascend, Intel, Moore Threads, Rebellions, MetaX).
Research Extensions: Incorporates advanced research ideas including TurboQuant, CacheBlend, CacheGen, and Cartridges.

Excerpt from official Github repository of the project:

LMCache is a KV cache management layer for LLM inference. It turns KV cache from a temporary state into reusable AI-native knowledge that can be stored persistently, reused across multiple serving engines, monitored with an observability stack, and transformed for better generation quality. As a result, LMCache reduces TTFT (time-to-first-token) and improves throughput, especially for long-context agentic, multi-turn conversation, and knowledge-augmented workloads (e.g., RAG).

LMCache is vendor-neutral. It can be used as a KV cache layer for a range of mainstream open-source serving engines, inference frameworks, hardware vendors, storage systems, and infrastructure providers. The vendor neutrality allows users to freely switch between serving engines and storage vendors, while reusing the stored KV caches.

Key features

Engine-independent deployment : LMCache, as a standalone daemon process, manages KV cache independently from the inference engine process, so that KV cache will not be lost even if the inference engine crashes (i.e., no fate-sharing with engines).
Persistent, tiered KV cache off and reuse : Move KV caches out of GPU memory into a tiered storage hierarchy spanning CPU memory, local storage, and remote backends, enabling reuse across requests, sessions, and engine instances to reduce repeated prefill computation and improve TTFT.
Production-level KV cache observability : LMCache provides a rich set of KV cache observability metrics, including typical Kubernetes metrics (health monitoring, performance diagnostics), KV-cache-specific metrics (request-level and token-level prefix cache hits, lifecycle, request-level KV cache performance), management metrics (user-specific usage), and more.
Pluggable storage and transport backends : Easily integrate remote storage and KV transfer backends through a unified interface, enabling KV cache off and sharing across storage providers. Through this interface, LMCache supports storage backends including CPU RAM, local disk (SSD), Redis/Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL, and GDS.
Non-prefix KV reuse : Extend KV reuse beyond prefix caching by reusing cached KV blocks at any position in the prompt. This leverages CacheBlend to selectively recompute tokens for quality recovery.
PD disaggregation and KV transfer : Support KV cache transfer from prefill workers to decode workers over NVLink, RDMA, or TCP through transport layers such as NIXL.
Pluggable KV transformation : A simple interface for researchers to write compression, token dropping, and custom serialization through a flexible SERDE interface.

LMCache is becoming an integral layer in the LLM inference ecosystem, with community-driven integration with serving engines, inference frameworks, hardware vendors, storage systems, and infrastructure providers:

#

Implementation

Disclaimer: Officially, LMCache targets server-side GPU inference engines (vLLM, SGLang, TRT-LLM) on Linux; official PyPI wheels require CUDA, and feature requests for Ollama/LM Studio support (e.g., issue #923) were closed as "not planned". However, for a mock test and rapid local prototyping, an LMCache-style framework named KVBridge was implemented to bring these concepts to my local test environment.

#

Application overview & purpose

KVBridge demonstrates the core value proposition of LMCache — a KV cache management layer for scalable LLM inference — on a developer laptop. LMCache itself targets server-side engines (vLLM, SGLang, TensorRT-LLM) and needs CUDA. KVBridge re-implements the LMCache concepts (chunked, content-addressed KV reuse; tiered storage; engine-independent keys) on top of the engines developers actually run: two local ones (llama.cpp, Ollama) and any OpenAI-compatible cloud service, all behind one pluggable engine registry.

What KVBridge is, and is not

KVBridge is an LMCache-style implementation. It is not the LMCache package. It optionally reuses LMCache's native TokenHasher when the real package can be built (scripts/install_lmcache.sh); otherwise a bit-compatible blake3 rolling hash is used.

#

System Architecture & Components

KVBridge mirrors LMCache's core mechanisms (rolling prefix hashing, tiered storage, and cache persistence) while adding an extensible engine registry to support local engines (llama.cpp, Ollama) and any OpenAI-compatible cloud provider.

#

Code Layout

The local application, uses Gradio for the UI implementation and is configurable to connect to multiple backends.

lmcache-test/
├── kvbridge/
│   ├── __init__.py
│   ├── __main__.py        # python -m kvbridge
│   ├── app.py             # Gradio GUI (registry-driven), AppState, main(), SIGTERM handler
│   ├── backends.py        # engine registry, LlamaCppBackend, OllamaBackend, CloudBackend, make_backend
│   ├── engine.py          # KVBridgeEngine: lookup, restore, compute, store
│   ├── store.py           # L1 RAM / L2 disk tiered store
│   ├── hashing.py         # chunking + chained hashes
│   ├── metrics.py         # counters, TTFT, hit rates
│   ├── charts.py          # Plotly figures
│   ├── scenarios.py       # demo scenario registry
│   └── config.py          # Settings (GUI/env/.env), presets, update_env_file
├── tests/                 # unit + live tests
├── scripts/               # setup, start, stop, test, cleanup, install_lmcache
├── Docs/                  # Quickstart, Architecture, Research, this file
├── input/  output/        # tracked via .gitkeep, contents ignored
├── requirements.txt
├── pytest.ini
├── .env.example           # every KVB_* variable, local and cloud examples
└── README.md              # includes the Apache-2.0 licence statement

#

Evaluation Modes

To evaluate performance gains, the architecture routes requests through three distinct operational modes:

cold : Disables native prompt caching (cache_prompt=false ) and wipes existing slot state, forcing a full prefill calculation on every request to establish a baseline. 2. native : Relies strictly on the engine's built-in, in-slot prefix caching (cache_prompt=true ). 3. lmcache : Drops in-slot state, performs chunk-hash lookups against the L1/L2 store, restores cached tensor state into the engine slot, pre-fills only the un-cached tail suffix, and saves new chunk snapshots upon completion.

Full KV State Export and Import (engine.py) : the primary pipeline manages KV tensor persistence using (only for mock test purpose)llama.cpp 's/slots/{id}?action=save|restore API:

""The cache-aware inference pipeline.

Request flow (llama.cpp, full integration)::

    prompt --tokenize--> tokens --chunk+hash--> [h0 h1 h2 ...]
                                                   |
                    store.lookup (longest prefix) <-+
                          |hit                |miss
            fetch snapshot (L1 RAM / L2 disk)  |
            restore into engine slot           |
                          +---------+----------+
                                    v
                 engine.generate (prefills only the *missing* suffix)
                                    v
                 save slot -> store.put (only when new chunks were computed)

Three execution modes make the benefit measurable:

``cold``    prefix reuse disabled - every request prefills the whole prompt.
``native``  the engine's own slot/prefix cache only (baseline a user has today).
``lmcache`` native cache is dropped first (modelling eviction by another
            session, a restart, ...) and the KV layer restores state instead.
"""
...

def _infer_full(self, prompt: str, rec: RequestRecord, n_predict: int, temperature: float) -> None:
    """llama.cpp path: real KV tensor save/restore handling."""
    b = self.backend
    tokens = b.tokenize(prompt)

    if rec.mode == "cold":
        b.reset_native_cache()
        self._fill(rec, b.generate(prompt, tokens, n_predict, False, temperature))
        return

    if rec.mode == "native":
        self._fill(rec, b.generate(prompt, tokens, n_predict, True, temperature))
        return

    hashes = self.hasher.chunk_hashes(tokens)
    rec.chunks_total = len(hashes)
    rec.chunk_status = self.store.chunk_status(hashes)

    b.reset_native_cache()                       
    match = self.store.lookup(hashes)
    rec.lmcache_match_tokens, rec.chunks_hit, rec.tier = match.tokens, match.chunks, match.tier

    if match.entry is not None:
        try:
            blob = self.store.fetch(match.entry)
            n = b.restore_state(blob)
            rec.restore_ms = (time.perf_counter() - t0) * 1000
        except (BackendError, KeyError, OSError) as exc:
            log.warning("Restore failed (%s); falling back to recompute", exc)
            rec.chunks_hit, rec.lmcache_match_tokens, rec.tier = 0, 0, "none"
            b.reset_native_cache()

    res = b.generate(prompt, tokens, n_predict, True, temperature)
    self._fill(rec, res)

    if len(hashes) > rec.chunks_hit:
        blob, n_saved = b.save_state()
        self.store.put(hashes, len(hashes) * self.s.chunk_size, blob,
                       note=f"{n_saved} tok snapshot")

Extensible Engine Registry (backends.py) : to prevent hardcoded conditionals across engine implementations, a centralized registry handles dynamic initialization:

"""Inference-engine adapters.

Three adapters share one small interface (:class:`Backend`) and are looked up
through a registry (:func:`register_engine`, :func:` resolve_engine`), so the
engine is chosen purely by configuration (``KVB_ENGINE`` or the GUI):

``LlamaCppBackend``
    Talks to ``llama-server`` (llama.cpp).  llama.cpp exposes the **slot
    save/restore API** (``POST /slots/{id}?action=save|restore|erase``) which
    serialises the *real KV tensors* of a sequence.  That is the integration
    point that lets an external cache layer (LMCache semantics) take KV out of
    the engine and inject it back -> **full integration**.

``OllamaBackend``
    Talks to the Ollama REST API.  Ollama deliberately hides KV state: there
    is no KV export/import and no tokenizer endpoint.  The adapter therefore
    works in **shadow mode** - the LMCache layer indexes/accounts for chunks
    and predicts reuse, while Ollama's own prefix cache does the real work and
    its ``prompt_eval_cached_count`` is used as ground truth.

``CloudBackend``
    Talks to any **OpenAI-compatible** HTTP API (OpenAI, Groq, OpenRouter,
    Together, Mistral, DeepSeek, vLLM, LM Studio, ...).  Hosted engines never
    expose their KV tensors either, so it also runs in **shadow mode**; the
    provider's own prompt-cache counter (``cached_tokens``) is the ground truth.
    Prompts leave the machine - the GUI says so explicitly.
"""

from __future__ import annotations

import json
import logging
import os
import re
import shutil
import signal
import subprocess
import time
import uuid
import zlib
from abc import ABC, abstractmethod
from dataclasses import dataclass
from pathlib import Path
from typing import Optional, Sequence
from urllib.parse import urlparse

import requests

from .config import CLOUD_API_STYLES, DEFAULT_LLAMA_SEARCH_PATHS, Settings
...

@dataclass(frozen=True)
class EngineSpec:
    key: str                      # Identifier (e.g., KVB_ENGINE)
    label: str                    # GUI Label
    cls: type[Backend]            # Adapter implementation class
    blurb: str = ""
    aliases: tuple[str, ...] = ()

_REGISTRY: dict[str, EngineSpec] = {}

def register_engine(spec: EngineSpec) -> EngineSpec:
    _REGISTRY[_norm(spec.key)] = spec
    return spec

def make_backend(kind: str, settings: Settings) -> Backend:
    return resolve_engine(kind).cls(settings)

register_engine(EngineSpec("llamacpp", "llama.cpp (local)", LlamaCppBackend,
                           "Full KV save/restore through llama-server slots.", ("llama",)))
register_engine(EngineSpec("ollama", "Ollama (local)", OllamaBackend,
                           "Shadow mode: Ollama hides its KV cache.", ()))
register_engine(EngineSpec("cloud", "Cloud / OpenAI-compatible API", CloudBackend,
                           "Shadow mode against hosted engines.", ("openai", "api")))

Chained Rolling Prefix Hasher (hashing.py) : token streams are packed into big-endian uint32 representations and processed sequentially:

"""Chunk-level rolling prefix hashing, compatible with LMCache.

LMCache splits a token sequence into fixed-size chunks (default 256 tokens)
and identifies each chunk by a *rolling prefix hash*::

    h_0 = blake3(prefix_seed        || pack(chunk_0))
    h_i = blake3(h_{i-1}            || pack(chunk_i))

where ``pack`` is the big-endian `` uint32`` encoding of the token ids
(``lmcache/v1/multiprocess/token_codec.py``).  Only *complete* chunks are
hashed; a trailing partial chunk is discarded - exactly like
``lmcache.v1.multiprocess.token_hasher.TokenHasher``.

This module re-implements that algorithm (so the app runs on macOS where the
full LMCache wheel cannot be installed) and can optionally delegate to the
real ``TokenHasher`` when the `` lmcache`` package is importable.  A unit test
verifies both produce identical digests.
"""

from __future__ import annotations

import hashlib
import logging
import struct
from typing import Protocol, Sequence

log = logging.getLogger(__name__)

try:  # blake3 is the algorithm LMCache uses
    import blake3 as _blake3

    def _digest(data_parts: Sequence[bytes]) -> bytes:
        h = _blake3.blake3()
        for part in data_parts:
            h.update(part)
        return h.digest()

    ALGORITHM = "blake3"
except ImportError:  # pragma: no cover - fallback keeps the app usable
    def _digest(data_parts: Sequence[bytes]) -> bytes:
        h = hashlib.sha256()
        for part in data_parts:
            h.update(part)
        return h.digest()

    ALGORITHM = "sha256 (blake3 missing; NOT LMCache compatible)"

def pack_tokens(tokens: Sequence[int]) -> bytes:
    """Pack token ids as big-endian ``uint32`` (LMCache wire format)."""
    return struct.pack(f">{len(tokens)}I", *tokens)

def namespace_seed(namespace: str) -> bytes:
    """Derive the initial prefix hash from a model/config namespace.

    KV tensors are only valid for the exact model, quantisation and context
    configuration that produced them, so the namespace is mixed into the
    root of every hash chain (LMCache keys also embed the model name).
    """
    return _digest([namespace.encode("utf-8")])

class Hasher(Protocol):
    chunk_size: int
    impl: str

    def chunk_hashes(self, tokens: Sequence[int]) -> list[bytes]: ...

class ChunkHasher:
    """Pure-Python implementation of LMCache's rolling chunk hashing."""

    impl = "kvbridge (LMCache-compatible blake3)"

    def __init__(self, chunk_size: int = 256, namespace: str = "") -> None:
        if chunk_size < 1:
            raise ValueError("chunk_size must be >= 1")
        self.chunk_size = chunk_size
        self.namespace = namespace
        self.seed = namespace_seed(namespace)

    def chunk_hashes(self, tokens: Sequence[int]) -> list[bytes]:
        """Return one rolling hash per *complete* chunk of ``tokens``."""
        n_complete = len(tokens) - len(tokens) % self.chunk_size
        prefix = self.seed
        out: list[bytes] = []
        for i in range(0, n_complete, self.chunk_size):
            prefix = _digest([prefix, pack_tokens(tokens[i : i + self.chunk_size])])
            out.append(prefix)
        return out

class LMCacheHasherAdapter:
    """Delegates to ``lmcache.v1.multiprocess.token_hasher.TokenHasher``."""

    def __init__(self, chunk_size: int = 256, namespace: str = "") -> None:
        from lmcache.v1.multiprocess.token_hasher import TokenHasher  # type: ignore

        self.chunk_size = chunk_size
        self.namespace = namespace
        self.seed = namespace_seed(namespace)
        self._hasher = TokenHasher(chunk_size=chunk_size, hash_algorithm="blake3")
        self.impl = "lmcache.TokenHasher (real LMCache)"

    def chunk_hashes(self, tokens: Sequence[int]) -> list[bytes]:
        return self._hasher.compute_chunk_hashes(list(tokens), prefix_hash=self.seed)

def make_hasher(chunk_size: int, namespace: str, prefer_lmcache: bool = False) -> Hasher:
    """Create a hasher, preferring the real LMCache one when requested."""
    if prefer_lmcache:
        try:
            return LMCacheHasherAdapter(chunk_size, namespace)
        except Exception as exc:  # ImportError or LMCache init problems
            log.warning("LMCache TokenHasher unavailable (%s); using built-in hasher", exc)
    return ChunkHasher(chunk_size, namespace)

def common_prefix_len(a: Sequence[int], b: Sequence[int]) -> int:
    """Length of the longest common prefix of two token sequences."""
    n = min(len(a), len(b))
    i = 0
    while i < n and a[i] == b[i]:
        i += 1
    return i

#

Configuration examples & usage scenarios

Pick one block for .env (or enter the same values in the GUI and press Connect).

.env — local, full integration (llama.cpp):

KVB_ENGINE=llamacpp
KVB_OLLAMA_MODEL=llama3.2:latest      # used to locate the GGUF blob
KVB_CTX_SIZE=8192
KVB_LLAMA_START_TIMEOUT=300           # slow disk? allow more time

.env — local, shadow mode (Ollama):

KVB_ENGINE=ollama
KVB_OLLAMA_URL=http://127.0.0.1:11434
KVB_OLLAMA_MODEL=llama3.2:latest
KVB_OLLAMA_KEEP_ALIVE=30m

.env — hosted provider (key stays in your shell):

KVB_ENGINE=cloud
KVB_CLOUD_PROVIDER=openai
KVB_CLOUD_MODEL=gpt-4o-mini
KVB_CLOUD_API_KEY_ENV=OPENAI_API_KEY  # export OPENAI_API_KEY=... before start.sh
KVB_CLOUD_TOKEN_BUDGET=200000         # spend guard

.env — self-hosted vLLM / any OpenAI-compatible server:

KVB_ENGINE=cloud
KVB_CLOUD_BASE_URL=https://llm.example.internal/v1
KVB_CLOUD_MODEL=meta-llama/Llama-3.1-8B-Instruct
KVB_CLOUD_API_KEY_ENV=INTERNAL_LLM_KEY
KVB_CLOUD_EXTRA_HEADERS={"X-Team": "search"}

.env — free smoke test of the cloud path (no account):

KVB_ENGINE=cloud
KVB_CLOUD_PROVIDER=ollama-local-openai   # http://127.0.0.1:11434/v1
KVB_CLOUD_MODEL=llama3.2:latest

Scenario	Engine	What to do	What you see
A. Maximum benefit, fully offline	llama.cpp	`KVB_ENGINE=llamacpp` , Connect, runDocument Q&A	First question prefills (seconds); later ones restore KV in tens of ms; Engine restart shows an L2 restore
B. Observe an existing Ollama setup	Ollama	`KVB_ENGINE=ollama` , Connect, runShared system prompt	Predicted reuse next to `prompt_eval_cached_count` ; SHADOW label
C. Measure a hosted provider's prefix cache	Cloud	Preset + model + key + token budget, Connect, run Shared system prompt with a long input document	`cached_tokens` reported by the provider per request;`cold` adds a nonce and shows no reuse
D. Try the cloud path for free	Cloud	Preset `ollama-local-openai` , model`llama3.2:latest`	Same GUI and metrics, no key, no cost
E. Switch engines without editing files	any	Change the engine radio, edit fields, Connect, optionally Save settings to .env	Status line states engine, location and FULL/SHADOW mode

Programmatic equivalent:

from kvbridge.config import Settings
from kvbridge.backends import make_backend
from kvbridge.engine import CachedEngine

s = Settings.from_env()                  # honours KVB_ENGINE and .env
assert not (problems := s.validate()), problems
engine = CachedEngine(s, make_backend(s.engine, s))
engine.start()

#

Final Disclaimer and findings after the tests

🚩 As noted earlier, this application was designed to demystify LMCache functionality and practical usage. Operating as a local, GPU-free implementation, it offers a purely educational pathway to observe firsthand how tiered caching and prefix reuse enhance inference efficiency.

Question	Answer
Does LMCache officially support Ollama / llama.cpp?	No Supported: vLLM, SGLang, TRT-LLM. Issue #923 (Ollama / LM Studio) was closed “not planned”.
Can the PyPI wheel run on macOS?	No Wheels are manylinux-only and require CUDA ( `CUDA_HOME not set` ).
Can LMCache's hasher be used standalone?	Partly With `NO_GPU_EXT=1` plus extra dependencies the native`TokenHasher` builds.
Can llama.cpp export/import KV state?	Yes `/slots/{id}?action=save
Can a hosted (cloud) API export/import KV state?	No Never. Providers run their own prefix cache and report {% raw %} `cached_tokens` , enabling the same accounting-onlyshadow mode .
Can Ollama export/import KV state?	No No public KV API. It does report `prompt_eval_cached_count` , enabling an accounting-onlyshadow mode .

#

Conclusion

Building a practical testbed around LMCache concepts provides valuable insights into modern prompt caching mechanics.

Key takeaways from the project include:

Measurable TTFT Reduction : In benchmark scenarios (e.g., ~1.9k-token document Q&A), average TTFT dropped from 4,554 ms (cold) to 999 ms (lmcache)—achieving a 4.6x overall speedup across a 12-request workload. On warm hits, TTFT fell as low as 11 ms (L1 RAM) and 13 ms (L2 Disk). 2. Persistence vs. Native In-Slot Caching : Whilenative engine caching can be faster on a single warm slot (e.g., 64 ms), the primary value of an LMCache-style architecture is capacity and persistence—snapshots survive process restarts, span arbitrary numbers of prefixes, and remain usable across multiple sessions. 3. Observability & Control : Through an intuitive Gradio GUI, automated test suites (over 180 unit/live tests), and robust SIGTERM lifecycle handling, developers can reliably test, visualize, and optimize prompt reuse strategies on any workstation.

Thanks for reading 🌒

#

Links

Code repository for this post: https://github.com/aairom/lmcache-test
LMCache repository: https://github.com/LMCache/LMCache

source & further reading

dev.to — original article What I learned building an AI tool to remove people from photos I built AiRap.me: turn one photo into a lip-synced AI rap video Optimizing Real-Time Computer Vision Inference Pipelines in Python for Edge Monitoring

Harnessing LMCache: Building a Pluggable KV-Cache Management & Testing Framework

Run your AI side-project on zahid.host