Show HN: Kvcachescope – Why Nvidia-smi is blind to vLLM KV cache leaks

Kvcachescope, a logical memory profiler and state inspector for PagedAttention inference engines such as vLLM and SGLang, exposes KV cache leaks that standard GPU profilers like nvidia-smi miss by hooking into the engine's BlockSpaceManager to provide real-time visibility into logical-to-physical block mapping and detect unreleased blocks. The tool, available on GitHub, offers a web UI, a 10Hz WebSocket stream, and CI integration with zero-zombie tolerance assertions, addressing stalls caused by ungraceful client disconnects or fragmented decode sequences that keep physical memory allocated.

A logical memory profiler and state inspector for PagedAttention inference engines vLLM, SGLang . Standard GPU profilers nvidia-smi , nsys , torch.cuda.memory allocated observe physical VRAM allocations at the PyTorch tensor level. They cannot inspect the internal logical block tables, virtual token indices, reference counts, or prefix caching radix trees maintained inside an inference engine's memory manager. When an ungraceful client disconnect occurs, or when fragmented decode sequences hold tail blocks without allocation activity, physical memory stays allocated. kvcachescope hooks directly into the engine's BlockSpaceManager to provide real-time visibility into the logical-to-physical block mapping, flag state divergence between HTTP sessions and backend GPU allocations, and detect unreleased blocks in CI pipelines. +---------------------------------------------+ | kvcachescope Web UI | | - 2D Physical Block Grid Matrix | | - Virtual Token - Block Table Visualizer | | - Hostage / Leaked Sequence Inspector | +----------------------+----------------------+ ^ 10Hz WebSocket | /ws/stream v +------------------------------------+ +-------------------------------------+ | Target LLM Engine | | kvcachescope Server | | vLLM / SGLang / Disaggregated | | FastAPI + Detached Observer Loop | | | +-------------------------------------+ | +------------------------------+ | | | | BlockSpaceManager | | | | | - allocate | | | | | - free | | Telemetry Hook | State Divergence | | - append slots |==+========================== | Checks | | - block tables | | | | +------------------------------+ | v | | +-------------------------------------+ | +------------------------------+ | | CI / Regression Runner | | | PrefixCachingAllocator | | | - Zero-zombie tolerance assertions | | | - radix tree refcounts | | | - Perfetto trace export .json | | +------------------------------+ | +-------------------------------------+ +------------------------------------+ Run the profiler with the built-in PagedAttention simulation engine: git clone https://github.com/brian-mwirigi/kvcachescope.git cd kvcachescope pip install -r requirements.txt python run.py Dashboard starts at http://localhost:8000 . Attach kvcachescope to an active vLLM engine instance: python from vllm import LLMEngine, EngineArgs from backend.vllm hook import attach vllm hook Initialize standard vLLM engine engine args = EngineArgs model="facebook/opt-125m", enable prefix caching=True engine = LLMEngine.from engine args engine args Attach observer hook runs in isolated daemon thread hook = attach vllm hook engine, port=8000 The hook instruments BlockSpaceManager.allocate , free , append slots , and samples block tables at 10Hz without modifying model forward passes. When a continuous batching cluster hits 98% VRAM utilization and stalls, nvidia-smi reports all memory as allocated by Python. Attach kvcachescope to the running engine: python from backend.vllm hook import attach vllm hook hook = attach vllm hook llm engine, port=8000 Open http://localhost:8000 to inspect the Hostage Block & Zombie Hunter . If an ungraceful client disconnect left physical blocks locked, identify the exact sequence ID and call the reclaim endpoint POST /api/diagnostics/reclaim to restore the free queue without restarting the model pod. Add leak regression assertions to your GitHub Actions test suite: - name: KV Cache Memory Leak Assertion run: | python run.py --ci-mode --duration-sec 60 --max-zombie-tolerance 0 --max-frag-tolerance 35.0 --report-json ci report.json If a PR introduces reference count leaks or orphaned block allocations, the job automatically fails with exit code 1 and exports detailed diagnostics. When serving short-output agent loops 1–3 output tokens , large physical block sizes cause high internal slack waste. Benchmark your target traffic: python benchmarks/benchmark redline.py --concurrency 50 --duration 30 Inspect the Internal Slack Waste % metric to evaluate whether switching from 32-token to 16-token or 8-token block sizes recovers VRAM capacity for higher batch concurrency. Export microsecond-precision block lifecycle traces: python backend/stress test suite.py --vector all --export-perfetto trace.json Load trace.json into ui.perfetto.dev https://ui.perfetto.dev/ alongside native torch.profiler CUDA traces VLLM TORCH PROFILER DIR to verify chronological alignment between PagedAttention block deallocations and physical CUDA memory frees. Launch on cloud GPUs with zero network tunnels using notebooks/KVCacheScope Live vLLM Colab.ipynb /brian-mwirigi/kvcachescope/blob/main/notebooks/KVCacheScope Live vLLM Colab.ipynb . The notebook applies nest asyncio and exposes the UI in a native window via output.serve kernel port as window 8000 .When an HTTP client connection terminates mid-generation proxy timeout, client abort , ASGI servers raise asyncio.CancelledError . In some engine configurations, the frontend marks the request terminated but fails to dispatch abort request across the IPC boundary to the GPU worker. The worker continues autoregressive token decoding often at ~7-8 tokens/sec until reaching max tokens . StateDivergenceDetector cross-correlates frontend session registries with backend physical block tables to flag active token generation on closed sessions. When gpu memory utilization is configured near physical capacity =0.99 alongside high max num seqs , preemption edge cases in the scheduler can cause the main generation loop to deadlock while VRAM remains 99% full. kvcachescope 's observer executes on a decoupled daemon thread with lock-free atomic snapshot swaps, allowing telemetry streaming and sequence starvation reporting to continue even if the engine scheduler hangs. In multimodal or long system prompt workloads, Python-level cyclical references around shared prefix nodes can prevent garbage collection from destroying wrapper objects upon request termination. C++ block destructors are never invoked, leaving physical blocks with ref count 0 indefinitely. PagedAttention allocates fixed-size physical blocks default: 16 tokens . In high-concurrency short-output workloads e.g., 1-token tool calls or routing classifications , allocating a full 16-token block for 1 token leaves 15 unused slots 93.7% internal slack waste . Run automated memory leak assertions in GitHub Actions or test suites: python run.py --ci-mode --duration-sec 30 --max-zombie-tolerance 0 --max-frag-tolerance 40.0 --report-json ci report.json - Returns exit code 0 if all assertions pass. - Returns exit code 1 if unreleased hostage blocks or fragmentation thresholds are breached. - Writes structured results to ci report.json . Execute the 5-vector failure matrix: python backend/stress test suite.py --vector all Individual vectors: python backend/stress test suite.py --vector abort divergence Client disconnect vs backend worker python backend/stress test suite.py --vector deadlock 99 99.1% VRAM watermark lockup python backend/stress test suite.py --vector prefix cycle Multimodal prefix refcount leaks python backend/stress test suite.py --vector speculative thrash Dual-model rollback micro-allocations python backend/stress test suite.py --vector hw abstraction CUDA, ROCm, OpenVINO, QAic Run multi-threaded client load to test GPU saturation without hitting the Python asyncio client bottleneck 50 streams : python benchmarks/benchmark redline.py --concurrency 50 --duration 10 --disable-log-requests Exports a microsecond-precision trace file perfetto redline trace.json that can be loaded directly into ui.perfetto.dev https://ui.perfetto.dev/ for side-by-side alignment with torch.profiler . A notebook demonstrating live vLLM profiling with nest asyncio and Colab's native window port forwarding is in notebooks/KVCacheScope Live vLLM Colab.ipynb /brian-mwirigi/kvcachescope/blob/main/notebooks/KVCacheScope Live vLLM Colab.ipynb . python import nest asyncio nest asyncio.apply from google.colab import output output.serve kernel port as window 8000 usage: run.py -h --ci-mode --duration-sec DURATION SEC --max-zombie-tolerance MAX ZOMBIE TOLERANCE --max-frag-tolerance MAX FRAG TOLERANCE --scenario SCENARIO --report-json REPORT JSON --stress-suite --stress-vector {abort divergence,deadlock 99,prefix cycle,speculative thrash,hw abstraction} --export-perfetto EXPORT PERFETTO --port PORT --host HOST --no-browser Apache-2.0