{"slug": "harnessing-lmcache-building-a-pluggable-kv-cache-management-testing-framework", "title": "Harnessing LMCache: Building a Pluggable KV-Cache Management & Testing Framework", "summary": "A developer built a test and implementation framework around LMCache, an open-source KV-cache management layer for LLM inference that stores and reuses attention key-value tensors outside the serving engine. LMCache chunks token streams into 256-token blocks hashed with blake3 for prefix reuse and tiers cache snapshots across GPU/CPU memory, local disk, and remote backends such as Redis and S3, with support for vLLM, SGLang, and TensorRT-LLM. The project aims to cut prefill recomputation and time-to-first-token for long-context RAG and agentic workloads.", "body_md": "LMCache test & implementation.\n\n# \n  Introduction\n\nAs Large Language Model (LLM) architectures scale and context windows expand to tens of thousands of tokens, prefill latency has become a major bottleneck in AI application performance. Processing long, repeated prompt prefixes—such as system prompts, core reference documents in Retrieval-Augmented Generation (RAG), and extensive agent chat histories—consumes substantial compute cycles and drives up time-to-first-token (TTFT).\n\n## \n  What is LMCache?\n\n[LMCache](https://github.com/LMCache/LMCache) is an open-source Key-Value (KV) cache management layer designed to solve this exact bottleneck. Rather than recomputing attention Key and Value tensors for every incoming request, LMCache captures, indexes, and stores these tensors outside the core inference engine, re-injecting them on demand.\n\nKey features of LMCache include:\n\n- Chunk-Level Prefix Reuse: Token streams are partitioned into fixed-size chunks (defaulting to 256 tokens). Each chunk is assigned a rolling prefix hash (using `blake3` ), guaranteeing that matching prefixes immediately hit cached states.\n- Tiered Storage Architecture: Cache snapshots flow seamlessly across device memory (GPU/CPU RAM), local disk (L2), and remote backends (e.g., Redis, S3, Mooncake, NIXL) with Least Recently Used (LRU) eviction policies.\n- Engine Independence: Cache snapshots survive engine restarts and slot evictions, allowing cross-instance sharing and disaggregated prefill/decode topologies.\n\n## \n  The LMCache Ecosystem\n\nThe LMCache ecosystem spans mainstream open-source projects, enterprise hardware vendors, and infrastructure orchestrators:\n\n- Inference Engines: Native support targets server-grade Linux/CUDA engines such as vLLM, SGLang, and TensorRT-LLM.\n- Orchestration Frameworks: Integrates with orchestrators like NVIDIA Dynamo, vLLM Production Stack, `llm-d` , and KServe.\n- Hardware & Storage Ecosystem: Compatible with storage vendors (Dell, Redis, DDN, Valkey, WEKA, VAST, Samsung, IBM Storage) and accelerator providers (NVIDIA, AMD, Ascend, Intel, Moore Threads, Rebellions, MetaX).\n- Research Extensions: Incorporates advanced research ideas including TurboQuant, CacheBlend, CacheGen, and Cartridges.\n\n**Excerpt from official Github repository of the project:**\n\nLMCache is a **KV cache management layer** for LLM inference. It turns KV cache from a temporary state into reusable *AI-native knowledge* that can be *stored* persistently, *reused* across multiple serving engines, *monitored* with an observability stack, and *transformed* for better generation quality. As a result, LMCache **reduces TTFT** (time-to-first-token) and **improves throughput**, especially for long-context agentic, multi-turn conversation, and knowledge-augmented workloads (e.g., RAG).\n\nLMCache is **vendor-neutral**. It can be used as a KV cache layer for a range of mainstream open-source serving engines, inference frameworks, hardware vendors, storage systems, and infrastructure providers. The vendor neutrality allows users to freely switch between serving engines and storage vendors, while reusing the stored KV caches.\n\n### \n  Key features\n\n- **Engine-independent deployment** : LMCache, as a standalone daemon process, manages KV cache independently from the inference engine process, so that KV cache will not be lost even if the inference engine crashes (i.e., no fate-sharing with engines).\n- **Persistent, tiered KV cache offloading and reuse** : Move KV caches out of GPU memory into a tiered storage hierarchy spanning CPU memory, local storage, and remote backends, enabling reuse across requests, sessions, and engine instances to reduce repeated prefill computation and improve TTFT.\n- **Production-level KV cache observability** : LMCache provides a rich set of KV cache observability metrics, including typical Kubernetes metrics (health monitoring, performance diagnostics), KV-cache-specific metrics (request-level and token-level prefix cache hits, lifecycle, request-level KV cache performance), management metrics (user-specific usage), and more.\n- **Pluggable storage and transport backends** : Easily integrate remote storage and KV transfer backends through a unified interface, enabling KV cache offloading and sharing across storage providers. Through this interface, LMCache supports storage backends including CPU RAM, local disk (SSD), Redis/Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL, and GDS.\n- **Non-prefix KV reuse** : Extend KV reuse beyond prefix caching by reusing cached KV blocks at any position in the prompt. This leverages CacheBlend to selectively recompute tokens for quality recovery.\n- **PD disaggregation and KV transfer** : Support KV cache transfer from prefill workers to decode workers over NVLink, RDMA, or TCP through transport layers such as NIXL.\n- **Pluggable KV transformation** : A simple interface for researchers to write compression, token dropping, and custom serialization through a flexible SERDE interface.\n\nLMCache is becoming an integral layer in the LLM inference *ecosystem*, with *community*-driven integration with serving engines, inference frameworks, hardware vendors, storage systems, and infrastructure providers:\n\n# \n  Implementation\n\n**Disclaimer: *Officially, LMCache targets server-side GPU inference engines (vLLM, SGLang, TRT-LLM) on Linux; official PyPI wheels require CUDA, and feature requests for Ollama/LM Studio support (e.g., issue #923) were closed as \"not planned\". However, for a mock test and rapid local prototyping, an LMCache-style framework named* *KVBridge* *was implemented to bring these concepts to my local test environment.***\n\n## \n  Application overview & purpose\n\nKVBridge demonstrates the core value proposition of [LMCache](https://github.com/LMCache/LMCache) — *a KV cache management layer for scalable LLM inference* — on a developer laptop. LMCache itself targets server-side engines (vLLM, SGLang, TensorRT-LLM) and needs CUDA. KVBridge **re-implements the LMCache concepts** (chunked, content-addressed KV reuse; tiered storage; engine-independent keys) on top of the engines developers actually run: two local ones (llama.cpp, Ollama) and any OpenAI-compatible cloud service, all behind one pluggable engine registry.\n\n**What KVBridge is, and is not**\n\nKVBridge is an LMCache-*style* implementation. It is **not** the LMCache package. It optionally reuses LMCache's native `TokenHasher` when the real package can be built (`scripts/install_lmcache.sh`); otherwise a bit-compatible `blake3` rolling hash is used.\n\n## \n  System Architecture & Components\n\nKVBridge mirrors LMCache's core mechanisms (rolling prefix hashing, tiered storage, and cache persistence) while adding an extensible engine registry to support local engines (`llama.cpp`, `Ollama`) and any OpenAI-compatible cloud provider.\n\n## \n  Code Layout\n\nThe local application, uses Gradio for the UI implementation and is configurable to connect to multiple backends.\n\n```\nlmcache-test/\n├── kvbridge/\n│   ├── __init__.py\n│   ├── __main__.py        # python -m kvbridge\n│   ├── app.py             # Gradio GUI (registry-driven), AppState, main(), SIGTERM handler\n│   ├── backends.py        # engine registry, LlamaCppBackend, OllamaBackend, CloudBackend, make_backend\n│   ├── engine.py          # KVBridgeEngine: lookup, restore, compute, store\n│   ├── store.py           # L1 RAM / L2 disk tiered store\n│   ├── hashing.py         # chunking + chained hashes\n│   ├── metrics.py         # counters, TTFT, hit rates\n│   ├── charts.py          # Plotly figures\n│   ├── scenarios.py       # demo scenario registry\n│   └── config.py          # Settings (GUI/env/.env), presets, update_env_file\n├── tests/                 # unit + live tests\n├── scripts/               # setup, start, stop, test, cleanup, install_lmcache\n├── Docs/                  # Quickstart, Architecture, Research, this file\n├── input/  output/        # tracked via .gitkeep, contents ignored\n├── requirements.txt\n├── pytest.ini\n├── .env.example           # every KVB_* variable, local and cloud examples\n└── README.md              # includes the Apache-2.0 licence statement\n```\n\n## \n  Evaluation Modes\n\nTo evaluate performance gains, the architecture routes requests through three distinct operational modes:\n\n1. \n**cold** : Disables native prompt caching (`cache_prompt=false` ) and wipes existing slot state, forcing a full prefill calculation on every request to establish a baseline.\n2. \n**native** : Relies strictly on the engine's built-in, in-slot prefix caching (`cache_prompt=true` ).\n3. \n**lmcache** : Drops in-slot state, performs chunk-hash lookups against the L1/L2 store, restores cached tensor state into the engine slot, pre-fills only the un-cached tail suffix, and saves new chunk snapshots upon completion.\n\n- \n**Full KV State Export and Import (`engine.py`)** : the primary pipeline manages KV tensor persistence using (only for mock test purpose)`llama.cpp` 's`/slots/{id}?action=save|restore` API:\n\n```\n\"\"The cache-aware inference pipeline.\n\nRequest flow (llama.cpp, full integration)::\n\n    prompt --tokenize--> tokens --chunk+hash--> [h0 h1 h2 ...]\n                                                   |\n                    store.lookup (longest prefix) <-+\n                          |hit                |miss\n            fetch snapshot (L1 RAM / L2 disk)  |\n            restore into engine slot           |\n                          +---------+----------+\n                                    v\n                 engine.generate (prefills only the *missing* suffix)\n                                    v\n                 save slot -> store.put (only when new chunks were computed)\n\nThree execution modes make the benefit measurable:\n\n``cold``    prefix reuse disabled - every request prefills the whole prompt.\n``native``  the engine's own slot/prefix cache only (baseline a user has today).\n``lmcache`` native cache is dropped first (modelling eviction by another\n            session, a restart, ...) and the KV layer restores state instead.\n\"\"\"\n...\n\ndef _infer_full(self, prompt: str, rec: RequestRecord, n_predict: int, temperature: float) -> None:\n    \"\"\"llama.cpp path: real KV tensor save/restore handling.\"\"\"\n    b = self.backend\n    tokens = b.tokenize(prompt)\n\n    if rec.mode == \"cold\":\n        b.reset_native_cache()\n        self._fill(rec, b.generate(prompt, tokens, n_predict, False, temperature))\n        return\n\n    if rec.mode == \"native\":\n        self._fill(rec, b.generate(prompt, tokens, n_predict, True, temperature))\n        return\n\n    # ---- LMCache Mode Pipeline\n    hashes = self.hasher.chunk_hashes(tokens)\n    rec.chunks_total = len(hashes)\n    rec.chunk_status = self.store.chunk_status(hashes)\n\n    # Drop native slot state to isolate LMCache retrieval benefit\n    b.reset_native_cache()                       \n    match = self.store.lookup(hashes)\n    rec.lmcache_match_tokens, rec.chunks_hit, rec.tier = match.tokens, match.chunks, match.tier\n\n    # Restore matching KV snapshot from L1 RAM or L2 Disk\n    if match.entry is not None:\n        try:\n            blob = self.store.fetch(match.entry)\n            n = b.restore_state(blob)\n            rec.restore_ms = (time.perf_counter() - t0) * 1000\n        except (BackendError, KeyError, OSError) as exc:\n            log.warning(\"Restore failed (%s); falling back to recompute\", exc)\n            rec.chunks_hit, rec.lmcache_match_tokens, rec.tier = 0, 0, \"none\"\n            b.reset_native_cache()\n\n    # Compute generation (prefills missing suffix only)\n    res = b.generate(prompt, tokens, n_predict, True, temperature)\n    self._fill(rec, res)\n\n    # Persist newly computed tail chunks into the Tiered Store\n    if len(hashes) > rec.chunks_hit:\n        blob, n_saved = b.save_state()\n        self.store.put(hashes, len(hashes) * self.s.chunk_size, blob,\n                       note=f\"{n_saved} tok snapshot\")\n```\n\n- \n**Extensible Engine Registry (`backends.py`)** : to prevent hardcoded conditionals across engine implementations, a centralized registry handles dynamic initialization:\n\n```\n\"\"\"Inference-engine adapters.\n\nThree adapters share one small interface (:class:`Backend`) and are looked up\nthrough a registry (:func:`register_engine`, :func:` resolve_engine`), so the\nengine is chosen purely by configuration (``KVB_ENGINE`` or the GUI):\n\n``LlamaCppBackend``\n    Talks to ``llama-server`` (llama.cpp).  llama.cpp exposes the **slot\n    save/restore API** (``POST /slots/{id}?action=save|restore|erase``) which\n    serialises the *real KV tensors* of a sequence.  That is the integration\n    point that lets an external cache layer (LMCache semantics) take KV out of\n    the engine and inject it back -> **full integration**.\n\n``OllamaBackend``\n    Talks to the Ollama REST API.  Ollama deliberately hides KV state: there\n    is no KV export/import and no tokenizer endpoint.  The adapter therefore\n    works in **shadow mode** - the LMCache layer indexes/accounts for chunks\n    and predicts reuse, while Ollama's own prefix cache does the real work and\n    its ``prompt_eval_cached_count`` is used as ground truth.\n\n``CloudBackend``\n    Talks to any **OpenAI-compatible** HTTP API (OpenAI, Groq, OpenRouter,\n    Together, Mistral, DeepSeek, vLLM, LM Studio, ...).  Hosted engines never\n    expose their KV tensors either, so it also runs in **shadow mode**; the\n    provider's own prompt-cache counter (``cached_tokens``) is the ground truth.\n    Prompts leave the machine - the GUI says so explicitly.\n\"\"\"\n\nfrom __future__ import annotations\n\nimport json\nimport logging\nimport os\nimport re\nimport shutil\nimport signal\nimport subprocess\nimport time\nimport uuid\nimport zlib\nfrom abc import ABC, abstractmethod\nfrom dataclasses import dataclass\nfrom pathlib import Path\nfrom typing import Optional, Sequence\nfrom urllib.parse import urlparse\n\nimport requests\n\nfrom .config import CLOUD_API_STYLES, DEFAULT_LLAMA_SEARCH_PATHS, Settings\n...\n\n@dataclass(frozen=True)\nclass EngineSpec:\n    key: str                      # Identifier (e.g., KVB_ENGINE)\n    label: str                    # GUI Label\n    cls: type[Backend]            # Adapter implementation class\n    blurb: str = \"\"\n    aliases: tuple[str, ...] = ()\n\n_REGISTRY: dict[str, EngineSpec] = {}\n\ndef register_engine(spec: EngineSpec) -> EngineSpec:\n    _REGISTRY[_norm(spec.key)] = spec\n    return spec\n\ndef make_backend(kind: str, settings: Settings) -> Backend:\n    return resolve_engine(kind).cls(settings)\n\n# Engine Registration\nregister_engine(EngineSpec(\"llamacpp\", \"llama.cpp (local)\", LlamaCppBackend,\n                           \"Full KV save/restore through llama-server slots.\", (\"llama\",)))\nregister_engine(EngineSpec(\"ollama\", \"Ollama (local)\", OllamaBackend,\n                           \"Shadow mode: Ollama hides its KV cache.\", ()))\nregister_engine(EngineSpec(\"cloud\", \"Cloud / OpenAI-compatible API\", CloudBackend,\n                           \"Shadow mode against hosted engines.\", (\"openai\", \"api\")))\n```\n\n- \n**Chained Rolling Prefix Hasher (`hashing.py`)** : token streams are packed into big-endian` uint32` representations and processed sequentially:\n\n```\n\"\"\"Chunk-level rolling prefix hashing, compatible with LMCache.\n\nLMCache splits a token sequence into fixed-size chunks (default 256 tokens)\nand identifies each chunk by a *rolling prefix hash*::\n\n    h_0 = blake3(prefix_seed        || pack(chunk_0))\n    h_i = blake3(h_{i-1}            || pack(chunk_i))\n\nwhere ``pack`` is the big-endian `` uint32`` encoding of the token ids\n(``lmcache/v1/multiprocess/token_codec.py``).  Only *complete* chunks are\nhashed; a trailing partial chunk is discarded - exactly like\n``lmcache.v1.multiprocess.token_hasher.TokenHasher``.\n\nThis module re-implements that algorithm (so the app runs on macOS where the\nfull LMCache wheel cannot be installed) and can optionally delegate to the\nreal ``TokenHasher`` when the `` lmcache`` package is importable.  A unit test\nverifies both produce identical digests.\n\"\"\"\n\nfrom __future__ import annotations\n\nimport hashlib\nimport logging\nimport struct\nfrom typing import Protocol, Sequence\n\nlog = logging.getLogger(__name__)\n\ntry:  # blake3 is the algorithm LMCache uses\n    import blake3 as _blake3\n\n    def _digest(data_parts: Sequence[bytes]) -> bytes:\n        h = _blake3.blake3()\n        for part in data_parts:\n            h.update(part)\n        return h.digest()\n\n    ALGORITHM = \"blake3\"\nexcept ImportError:  # pragma: no cover - fallback keeps the app usable\n    def _digest(data_parts: Sequence[bytes]) -> bytes:\n        h = hashlib.sha256()\n        for part in data_parts:\n            h.update(part)\n        return h.digest()\n\n    ALGORITHM = \"sha256 (blake3 missing; NOT LMCache compatible)\"\n\ndef pack_tokens(tokens: Sequence[int]) -> bytes:\n    \"\"\"Pack token ids as big-endian ``uint32`` (LMCache wire format).\"\"\"\n    return struct.pack(f\">{len(tokens)}I\", *tokens)\n\ndef namespace_seed(namespace: str) -> bytes:\n    \"\"\"Derive the initial prefix hash from a model/config namespace.\n\n    KV tensors are only valid for the exact model, quantisation and context\n    configuration that produced them, so the namespace is mixed into the\n    root of every hash chain (LMCache keys also embed the model name).\n    \"\"\"\n    return _digest([namespace.encode(\"utf-8\")])\n\nclass Hasher(Protocol):\n    chunk_size: int\n    impl: str\n\n    def chunk_hashes(self, tokens: Sequence[int]) -> list[bytes]: ...\n\nclass ChunkHasher:\n    \"\"\"Pure-Python implementation of LMCache's rolling chunk hashing.\"\"\"\n\n    impl = \"kvbridge (LMCache-compatible blake3)\"\n\n    def __init__(self, chunk_size: int = 256, namespace: str = \"\") -> None:\n        if chunk_size < 1:\n            raise ValueError(\"chunk_size must be >= 1\")\n        self.chunk_size = chunk_size\n        self.namespace = namespace\n        self.seed = namespace_seed(namespace)\n\n    def chunk_hashes(self, tokens: Sequence[int]) -> list[bytes]:\n        \"\"\"Return one rolling hash per *complete* chunk of ``tokens``.\"\"\"\n        n_complete = len(tokens) - len(tokens) % self.chunk_size\n        prefix = self.seed\n        out: list[bytes] = []\n        for i in range(0, n_complete, self.chunk_size):\n            prefix = _digest([prefix, pack_tokens(tokens[i : i + self.chunk_size])])\n            out.append(prefix)\n        return out\n\nclass LMCacheHasherAdapter:\n    \"\"\"Delegates to ``lmcache.v1.multiprocess.token_hasher.TokenHasher``.\"\"\"\n\n    def __init__(self, chunk_size: int = 256, namespace: str = \"\") -> None:\n        from lmcache.v1.multiprocess.token_hasher import TokenHasher  # type: ignore\n\n        self.chunk_size = chunk_size\n        self.namespace = namespace\n        self.seed = namespace_seed(namespace)\n        self._hasher = TokenHasher(chunk_size=chunk_size, hash_algorithm=\"blake3\")\n        self.impl = \"lmcache.TokenHasher (real LMCache)\"\n\n    def chunk_hashes(self, tokens: Sequence[int]) -> list[bytes]:\n        return self._hasher.compute_chunk_hashes(list(tokens), prefix_hash=self.seed)\n\ndef make_hasher(chunk_size: int, namespace: str, prefer_lmcache: bool = False) -> Hasher:\n    \"\"\"Create a hasher, preferring the real LMCache one when requested.\"\"\"\n    if prefer_lmcache:\n        try:\n            return LMCacheHasherAdapter(chunk_size, namespace)\n        except Exception as exc:  # ImportError or LMCache init problems\n            log.warning(\"LMCache TokenHasher unavailable (%s); using built-in hasher\", exc)\n    return ChunkHasher(chunk_size, namespace)\n\ndef common_prefix_len(a: Sequence[int], b: Sequence[int]) -> int:\n    \"\"\"Length of the longest common prefix of two token sequences.\"\"\"\n    n = min(len(a), len(b))\n    i = 0\n    while i < n and a[i] == b[i]:\n        i += 1\n    return i\n```\n\n## \n  Configuration examples & usage scenarios\n\nPick one block for `.env` (or enter the same values in the GUI and press *Connect*).\n\n- \n**.env — local, full integration (llama.cpp):**\n\n```\nKVB_ENGINE=llamacpp\nKVB_OLLAMA_MODEL=llama3.2:latest      # used to locate the GGUF blob\nKVB_CTX_SIZE=8192\nKVB_LLAMA_START_TIMEOUT=300           # slow disk? allow more time\n```\n\n- \n**.env — local, shadow mode (Ollama):**\n\n```\nKVB_ENGINE=ollama\nKVB_OLLAMA_URL=http://127.0.0.1:11434\nKVB_OLLAMA_MODEL=llama3.2:latest\nKVB_OLLAMA_KEEP_ALIVE=30m\n```\n\n- \n**.env — hosted provider (key stays in your shell):**\n\n```\nKVB_ENGINE=cloud\nKVB_CLOUD_PROVIDER=openai\nKVB_CLOUD_MODEL=gpt-4o-mini\nKVB_CLOUD_API_KEY_ENV=OPENAI_API_KEY  # export OPENAI_API_KEY=... before start.sh\nKVB_CLOUD_TOKEN_BUDGET=200000         # spend guard\n```\n\n- \n**.env — self-hosted vLLM / any OpenAI-compatible server:**\n\n```\nKVB_ENGINE=cloud\nKVB_CLOUD_BASE_URL=https://llm.example.internal/v1\nKVB_CLOUD_MODEL=meta-llama/Llama-3.1-8B-Instruct\nKVB_CLOUD_API_KEY_ENV=INTERNAL_LLM_KEY\nKVB_CLOUD_EXTRA_HEADERS={\"X-Team\": \"search\"}\n```\n\n- \n**.env — free smoke test of the cloud path (no account):**\n\n```\nKVB_ENGINE=cloud\nKVB_CLOUD_PROVIDER=ollama-local-openai   # http://127.0.0.1:11434/v1\nKVB_CLOUD_MODEL=llama3.2:latest\n```\n\n| Scenario | Engine | What to do | What you see | \n|---|---|---|---|\n| A. Maximum benefit, fully offline | llama.cpp | `KVB_ENGINE=llamacpp` , Connect, run*Document Q&A* | First question prefills (seconds); later ones restore KV in tens of ms; *Engine restart* shows an L2 restore | \n| B. Observe an existing Ollama setup | Ollama | `KVB_ENGINE=ollama` , Connect, run*Shared system prompt* | Predicted reuse next to `prompt_eval_cached_count` ; SHADOW label | \n| C. Measure a hosted provider's prefix cache | Cloud | Preset + model + key + token budget, Connect, run *Shared system prompt* with a long input document | `cached_tokens` reported by the provider per request;`cold` adds a nonce and shows no reuse | \n| D. Try the cloud path for free | Cloud | Preset `ollama-local-openai` , model`llama3.2:latest` | Same GUI and metrics, no key, no cost | \n| E. Switch engines without editing files | any | Change the engine radio, edit fields, Connect, optionally *Save settings to .env* | Status line states engine, location and FULL/SHADOW mode | \n\n- \n**Programmatic equivalent:**\n\n``` python\nfrom kvbridge.config import Settings\nfrom kvbridge.backends import make_backend\nfrom kvbridge.engine import CachedEngine\n\ns = Settings.from_env()                  # honours KVB_ENGINE and .env\nassert not (problems := s.validate()), problems\nengine = CachedEngine(s, make_backend(s.engine, s))\nengine.start()\n```\n\n## \n  Final Disclaimer and findings after the tests\n\n🚩    **As noted earlier, this application was designed to demystify LMCache functionality and practical usage. Operating as a local, GPU-free implementation, it offers a purely educational pathway to observe firsthand how tiered caching and prefix reuse enhance inference efficiency.**\n\n| Question | Answer | \n|---|---|\n| Does LMCache officially support Ollama / llama.cpp? | No Supported: vLLM, SGLang, TRT-LLM. Issue #923 (Ollama / LM Studio) was closed “not planned”. | \n| Can the PyPI wheel run on macOS? | No Wheels are manylinux-only and require CUDA ( `CUDA_HOME not set` ). | \n| Can LMCache's hasher be used standalone? | Partly With `NO_GPU_EXT=1` plus extra dependencies the native`TokenHasher` builds. | \n| Can llama.cpp export/import KV state? | Yes `/slots/{id}?action=save | \n| Can a hosted (cloud) API export/import KV state? | No Never. Providers run their own prefix cache and report {% raw %} `cached_tokens` , enabling the same accounting-only*shadow mode* . | \n| Can Ollama export/import KV state? | No No public KV API. It does report `prompt_eval_cached_count` , enabling an accounting-only*shadow mode* . | \n\n# \n  Conclusion\n\nBuilding a practical testbed around LMCache concepts provides valuable insights into modern prompt caching mechanics.\n\nKey takeaways from the project include:\n\n1. \n**Measurable TTFT Reduction** : In benchmark scenarios (e.g., ~1.9k-token document Q&A), average TTFT dropped from 4,554 ms (cold) to 999 ms (lmcache)—achieving a 4.6x overall speedup across a 12-request workload. On warm hits, TTFT fell as low as 11 ms (L1 RAM) and 13 ms (L2 Disk).\n2. \n**Persistence vs. Native In-Slot Caching** : While`native` engine caching can be faster on a single warm slot (e.g., 64 ms), the primary value of an LMCache-style architecture is capacity and persistence—snapshots survive process restarts, span arbitrary numbers of prefixes, and remain usable across multiple sessions.\n3. \n**Observability & Control** : Through an intuitive Gradio GUI, automated test suites (over 180 unit/live tests), and robust SIGTERM lifecycle handling, developers can reliably test, visualize, and optimize prompt reuse strategies on any workstation.\n\n**Thanks for reading 🌒**\n\n# \n  Links\n\n- Code repository for this post: [https://github.com/aairom/lmcache-test](https://github.com/aairom/lmcache-test)\n- LMCache repository: [https://github.com/LMCache/LMCache](https://github.com/LMCache/LMCache)", "url": "https://wpnews.pro/news/harnessing-lmcache-building-a-pluggable-kv-cache-management-testing-framework", "canonical_source": "https://dev.to/aairom/harnessing-lmcache-building-a-pluggable-kv-cache-management-testing-framework-5fej", "published_at": "2026-11-10 08:17:51+00:00", "updated_at": "2026-11-10 08:22:02.870837+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "mlops", "ai-tools", "ai-agents"], "entities": ["LMCache", "vLLM", "SGLang", "TensorRT-LLM", "Redis", "NVIDIA", "AMD", "KServe"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/harnessing-lmcache-building-a-pluggable-kv-cache-management-testing-framework", "markdown": "https://wpnews.pro/news/harnessing-lmcache-building-a-pluggable-kv-cache-management-testing-framework.md", "text": "https://wpnews.pro/news/harnessing-lmcache-building-a-pluggable-kv-cache-management-testing-framework.txt", "jsonld": "https://wpnews.pro/news/harnessing-lmcache-building-a-pluggable-kv-cache-management-testing-framework.jsonld"}}