{"slug": "show-hn-kvcachescope-why-nvidia-smi-is-blind-to-vllm-kv-cache-leaks", "title": "Show HN: Kvcachescope – Why Nvidia-smi is blind to vLLM KV cache leaks", "summary": "Kvcachescope, a logical memory profiler and state inspector for PagedAttention inference engines such as vLLM and SGLang, exposes KV cache leaks that standard GPU profilers like nvidia-smi miss by hooking into the engine's BlockSpaceManager to provide real-time visibility into logical-to-physical block mapping and detect unreleased blocks. The tool, available on GitHub, offers a web UI, a 10Hz WebSocket stream, and CI integration with zero-zombie tolerance assertions, addressing stalls caused by ungraceful client disconnects or fragmented decode sequences that keep physical memory allocated.", "body_md": "A logical memory profiler and state inspector for PagedAttention inference engines (vLLM, SGLang).\n\nStandard GPU profilers (`nvidia-smi`\n\n, `nsys`\n\n, `torch.cuda.memory_allocated()`\n\n) observe physical VRAM allocations at the PyTorch tensor level. They cannot inspect the internal logical block tables, virtual token indices, reference counts, or prefix caching radix trees maintained inside an inference engine's memory manager.\n\nWhen an ungraceful client disconnect occurs, or when fragmented decode sequences hold tail blocks without allocation activity, physical memory stays allocated. `kvcachescope`\n\nhooks directly into the engine's `BlockSpaceManager`\n\nto provide real-time visibility into the logical-to-physical block mapping, flag state divergence between HTTP sessions and backend GPU allocations, and detect unreleased blocks in CI pipelines.\n\n```\n                                      +---------------------------------------------+\n                                      |            kvcachescope Web UI              |\n                                      |  - 2D Physical Block Grid Matrix            |\n                                      |  - Virtual Token -> Block Table Visualizer  |\n                                      |  - Hostage / Leaked Sequence Inspector      |\n                                      +----------------------+----------------------+\n                                                             ^\n                                              10Hz WebSocket | /ws/stream\n                                                             v\n+------------------------------------+        +-------------------------------------+\n|        Target LLM Engine           |        |         kvcachescope Server         |\n|  (vLLM / SGLang / Disaggregated)   |        |  (FastAPI + Detached Observer Loop) |\n|                                    |        +-------------------------------------+\n|  +------------------------------+  |                           |\n|  |     BlockSpaceManager        |  |                           |\n|  |  - allocate()                |  |                           |\n|  |  - free()                    |  | Telemetry Hook            | State Divergence\n|  |  - append_slots()            |==+==========================>| Checks\n|  |  - block_tables              |  |                           |\n|  +------------------------------+  |                           v\n|                                    |        +-------------------------------------+\n|  +------------------------------+  |        |      CI / Regression Runner         |\n|  |   PrefixCachingAllocator     |  |        |  - Zero-zombie tolerance assertions |\n|  |  - radix_tree refcounts      |  |        |  - Perfetto trace export (.json)    |\n|  +------------------------------+  |        +-------------------------------------+\n+------------------------------------+\n```\n\nRun the profiler with the built-in PagedAttention simulation engine:\n\n```\ngit clone https://github.com/brian-mwirigi/kvcachescope.git\ncd kvcachescope\npip install -r requirements.txt\npython run.py\n```\n\nDashboard starts at `http://localhost:8000`\n\n.\n\nAttach `kvcachescope`\n\nto an active vLLM engine instance:\n\n``` python\nfrom vllm import LLMEngine, EngineArgs\nfrom backend.vllm_hook import attach_vllm_hook\n\n# Initialize standard vLLM engine\nengine_args = EngineArgs(model=\"facebook/opt-125m\", enable_prefix_caching=True)\nengine = LLMEngine.from_engine_args(engine_args)\n\n# Attach observer hook (runs in isolated daemon thread)\nhook = attach_vllm_hook(engine, port=8000)\n```\n\nThe hook instruments `BlockSpaceManager.allocate()`\n\n, `free()`\n\n, `append_slots()`\n\n, and samples `block_tables`\n\nat 10Hz without modifying model forward passes.\n\nWhen a continuous batching cluster hits 98% VRAM utilization and stalls, `nvidia-smi`\n\nreports all memory as allocated by Python. Attach `kvcachescope`\n\nto the running engine:\n\n``` python\nfrom backend.vllm_hook import attach_vllm_hook\nhook = attach_vllm_hook(llm_engine, port=8000)\n```\n\nOpen `http://localhost:8000`\n\nto inspect the **Hostage Block & Zombie Hunter**. If an ungraceful client disconnect left physical blocks locked, identify the exact sequence ID and call the reclaim endpoint (`POST /api/diagnostics/reclaim`\n\n) to restore the free queue without restarting the model pod.\n\nAdd leak regression assertions to your GitHub Actions test suite:\n\n```\n- name: KV Cache Memory Leak Assertion\n  run: |\n    python run.py --ci-mode --duration-sec 60 --max-zombie-tolerance 0 --max-frag-tolerance 35.0 --report-json ci_report.json\n```\n\nIf a PR introduces reference count leaks or orphaned block allocations, the job automatically fails with exit code `1`\n\nand exports detailed diagnostics.\n\nWhen serving short-output agent loops (1–3 output tokens), large physical block sizes cause high internal slack waste. Benchmark your target traffic:\n\n```\npython benchmarks/benchmark_redline.py --concurrency 50 --duration 30\n```\n\nInspect the **Internal Slack Waste %** metric to evaluate whether switching from 32-token to 16-token or 8-token block sizes recovers VRAM capacity for higher batch concurrency.\n\nExport microsecond-precision block lifecycle traces:\n\n```\npython backend/stress_test_suite.py --vector all --export-perfetto trace.json\n```\n\nLoad `trace.json`\n\ninto [ui.perfetto.dev](https://ui.perfetto.dev/) alongside native `torch.profiler`\n\nCUDA traces (`VLLM_TORCH_PROFILER_DIR`\n\n) to verify chronological alignment between PagedAttention block deallocations and physical CUDA memory frees.\n\nLaunch on cloud GPUs with zero network tunnels using [ notebooks/KVCacheScope_Live_vLLM_Colab.ipynb](/brian-mwirigi/kvcachescope/blob/main/notebooks/KVCacheScope_Live_vLLM_Colab.ipynb). The notebook applies\n\n`nest_asyncio`\n\nand exposes the UI in a native window via `output.serve_kernel_port_as_window(8000)`\n\n.When an HTTP client connection terminates mid-generation (proxy timeout, client abort), ASGI servers raise `asyncio.CancelledError`\n\n. In some engine configurations, the frontend marks the request terminated but fails to dispatch `abort_request()`\n\nacross the IPC boundary to the GPU worker.\n\nThe worker continues autoregressive token decoding (often at ~7-8 tokens/sec) until reaching `max_tokens`\n\n. `StateDivergenceDetector`\n\ncross-correlates frontend session registries with backend physical block tables to flag active token generation on closed sessions.\n\nWhen `gpu_memory_utilization`\n\nis configured near physical capacity (>=0.99) alongside high `max_num_seqs`\n\n, preemption edge cases in the scheduler can cause the main generation loop to deadlock while VRAM remains 99% full.\n\n`kvcachescope`\n\n's observer executes on a decoupled daemon thread with lock-free atomic snapshot swaps, allowing telemetry streaming and sequence starvation reporting to continue even if the engine scheduler hangs.\n\nIn multimodal or long system prompt workloads, Python-level cyclical references around shared prefix nodes can prevent garbage collection from destroying wrapper objects upon request termination. C++ block destructors are never invoked, leaving physical blocks with `ref_count > 0`\n\nindefinitely.\n\nPagedAttention allocates fixed-size physical blocks (default: 16 tokens). In high-concurrency short-output workloads (e.g., 1-token tool calls or routing classifications), allocating a full 16-token block for 1 token leaves 15 unused slots (93.7% internal slack waste).\n\nRun automated memory leak assertions in GitHub Actions or test suites:\n\n```\npython run.py --ci-mode --duration-sec 30 --max-zombie-tolerance 0 --max-frag-tolerance 40.0 --report-json ci_report.json\n```\n\n- Returns exit code\n`0`\n\nif all assertions pass. - Returns exit code\n`1`\n\nif unreleased hostage blocks or fragmentation thresholds are breached. - Writes structured results to\n`ci_report.json`\n\n.\n\nExecute the 5-vector failure matrix:\n\n```\npython backend/stress_test_suite.py --vector all\n```\n\nIndividual vectors:\n\n`python backend/stress_test_suite.py --vector abort_divergence`\n\n(Client disconnect vs backend worker)`python backend/stress_test_suite.py --vector deadlock_99`\n\n(99.1% VRAM watermark lockup)`python backend/stress_test_suite.py --vector prefix_cycle`\n\n(Multimodal prefix refcount leaks)`python backend/stress_test_suite.py --vector speculative_thrash`\n\n(Dual-model rollback micro-allocations)`python backend/stress_test_suite.py --vector hw_abstraction`\n\n(CUDA, ROCm, OpenVINO, QAic)\n\nRun multi-threaded client load to test GPU saturation without hitting the Python asyncio client bottleneck (>50 streams):\n\n```\npython benchmarks/benchmark_redline.py --concurrency 50 --duration 10 --disable-log-requests\n```\n\nExports a microsecond-precision trace file (`perfetto_redline_trace.json`\n\n) that can be loaded directly into [ui.perfetto.dev](https://ui.perfetto.dev/) for side-by-side alignment with `torch.profiler`\n\n.\n\nA notebook demonstrating live vLLM profiling with `nest_asyncio`\n\nand Colab's native window port forwarding is in [ notebooks/KVCacheScope_Live_vLLM_Colab.ipynb](/brian-mwirigi/kvcachescope/blob/main/notebooks/KVCacheScope_Live_vLLM_Colab.ipynb).\n\n``` python\nimport nest_asyncio\nnest_asyncio.apply()\n\nfrom google.colab import output\noutput.serve_kernel_port_as_window(8000)\nusage: run.py [-h] [--ci-mode] [--duration-sec DURATION_SEC]\n              [--max-zombie-tolerance MAX_ZOMBIE_TOLERANCE]\n              [--max-frag-tolerance MAX_FRAG_TOLERANCE]\n              [--scenario SCENARIO] [--report-json REPORT_JSON]\n              [--stress-suite] [--stress-vector {abort_divergence,deadlock_99,prefix_cycle,speculative_thrash,hw_abstraction}]\n              [--export-perfetto EXPORT_PERFETTO] [--port PORT]\n              [--host HOST] [--no-browser]\n```\n\nApache-2.0", "url": "https://wpnews.pro/news/show-hn-kvcachescope-why-nvidia-smi-is-blind-to-vllm-kv-cache-leaks", "canonical_source": "https://github.com/brian-mwirigi/kvcachescope", "published_at": "2026-08-14 23:11:14+00:00", "updated_at": "2026-08-14 23:41:32.549472+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure", "ai-tools"], "entities": ["Kvcachescope", "vLLM", "SGLang", "BlockSpaceManager", "PrefixCachingAllocator", "FastAPI", "GitHub", "Brian Mwirigi"], "alternates": {"html": "https://wpnews.pro/news/show-hn-kvcachescope-why-nvidia-smi-is-blind-to-vllm-kv-cache-leaks", "markdown": "https://wpnews.pro/news/show-hn-kvcachescope-why-nvidia-smi-is-blind-to-vllm-kv-cache-leaks.md", "text": "https://wpnews.pro/news/show-hn-kvcachescope-why-nvidia-smi-is-blind-to-vllm-kv-cache-leaks.txt", "jsonld": "https://wpnews.pro/news/show-hn-kvcachescope-why-nvidia-smi-is-blind-to-vllm-kv-cache-leaks.jsonld"}}