{"slug": "show-hn-sdcguard-inspect-vllm-kv-checksum-coverage-per-request", "title": "Show HN: SDCGuard – Inspect vLLM KV-checksum coverage per request", "summary": "Hari Krishna Kancharla released sdcguard, an MIT-licensed tool that checks whether vLLM's proposed KV-cache checksum actually compared each request's full prompt, after reproducing vLLM bug #61229 on an H100 where all 8 unfixed runs returned wrong answers with HTTP 200. In testing vLLM's unmerged checksum PR #62211, the checksum passed on all 153 requests but only 9 had their entire prompt compared, because the rest reused cache checked once on arrival — a behavior sdcguard reports as HOLD with exit code 2. Backporting the proposed fix PR #62065 made all 5 replays correct, and sdcguard currently supports one setup: the checksum PR at two pinned commits, prefill and decode on one H100 over NIXL with Qwen3-0.6B.", "body_md": "Release tests for LLM inference servers.\n\nAn inference server can send back a wrong answer with a normal HTTP 200. sdcguard shows what the server's own integrity checks covered for each request, so you can see it before you ship an engine upgrade.\n\nI reproduced vLLM bug [#61229](https://github.com/vllm-project/vllm/issues/61229) on an H100. The decode server\nreads KV blocks the prefill server already freed and reused, and I forced the timing with a delay. Asked to repeat\nthe secret word PLUM, one run answered \"M. Repeat the secret word...\" with a 200. All 8 unfixed runs came back\nwrong, and sdcguard flagged the stale read in every one. With the proposed fix\n([#62065](https://github.com/vllm-project/vllm/pull/62065)) backported, all 5 replays were right.\n[Saved results](https://github.com/hari-kancharla/sdcguard/blob/main/results/61229_saved).\n\nThen I tested vLLM's proposed KV checksum ([#62211](https://github.com/vllm-project/vllm/pull/62211), not merged\nyet). In a test where requests share most of their prompt, the checksum passed on all 153 requests, but only 9 had\ntheir whole prompt compared. The rest reused cache that was checked once, when it first arrived. That's by design,\nand SGLang does the same. But it means a pass doesn't tell you how much of the prompt was checked. I\ngot the same result on two builds of the PR and posted the numbers there.\n\n```\ngit clone https://github.com/hari-kancharla/sdcguard\ncd sdcguard\npython3 -m venv .venv\n. .venv/bin/activate\npip install --no-deps .\nsdcguard gate tests/data/checksum_owner_3a2ad847 --require checksum\n```\n\nThis replays the Oct 30 H100 run on your machine. No GPU needed, just Python 3.10 or newer.\n\n```\nHOLD | required: checksum\nselected recorded client requests only; no deployment or isolation guarantee\n9 / 153 selected requests passed\n...\nsgc-3a2ad847b9af4c76ae55aee4111c0258-0-1-35-d | HOLD | cached prefix not covered by checksum; checksum:cached prefix not covered by checksum\n  Prompt tokens: 1451; transferred tokens checked: 27; decode cache reused unchecked: 1424; prefill cache reused: 1424\n```\n\nIt exits 2 on HOLD and 0 on a pass. Two more commands replay saved runs:\n\n- `sdcguard transfer-check tests/data/checksum_owner_3a2ad847` checks whether every transfer was compared. It passes.\n- `sdcguard transfer-compare tests/data/checksum_owner_9b42f5ab tests/data/checksum_owner_b340ee03` compares two\nbuilds of the PR. 153 transfers kept their check and 0 lost it. It still exits 2 with HOLD, because the same\nrequests stay partly checked.\n\n``` php\nflowchart LR\n  P[Prefill server] -->|\"transfer, compared\"| D[Decode server]\n  K[(Decode prefix cache)] -->|\"reused, not compared\"| D\n  D -.->|\"checksum records\"| G[sdcguard gate]\n```\n\nA vLLM plugin records each checksum call and which tokens it compared. `sdcguard gate` matches those records to\nclient requests offline and passes a request only if its whole prompt was compared. The full method, recording setup\nand limits are in [docs/CHECKSUM_GATE.md](https://github.com/hari-kancharla/sdcguard/blob/main/docs/CHECKSUM_GATE.md).\n\nIt supports one vLLM setup today: the checksum PR at two pinned commits, prefill and decode on one H100 over NIXL, with Qwen3-0.6B. Recording a new run still needs that setup prepared by hand. Next is fault injection, then a self-service setup, then SGLang and a CI gate.\n\nsdcguard started as a GPU math checker. On a rented H200 split with NVIDIA MPS, it caught wrong results from a\nknown cuBLAS bug while NVIDIA DCGM and gpu-burn passed, because their tests never ran the broken code path\n([head-to-head](https://github.com/hari-kancharla/sdcguard/blob/main/results/h2h/h200_20261020_060928.md)). That version is on PyPI as\n`pip install sdcguard` (0.4.0).\n\nMIT license. Built by Hari Krishna Kancharla.", "url": "https://wpnews.pro/news/show-hn-sdcguard-inspect-vllm-kv-checksum-coverage-per-request", "canonical_source": "https://github.com/hari-kancharla/sdcguard", "published_at": "2026-11-02 17:47:12+00:00", "updated_at": "2026-11-02 17:52:42.187420+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops", "ai-safety"], "entities": ["sdcguard", "Hari Krishna Kancharla", "vLLM", "SGLang", "NVIDIA", "Qwen3-0.6B", "H100", "NIXL"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-sdcguard-inspect-vllm-kv-checksum-coverage-per-request", "markdown": "https://wpnews.pro/news/show-hn-sdcguard-inspect-vllm-kv-checksum-coverage-per-request.md", "text": "https://wpnews.pro/news/show-hn-sdcguard-inspect-vllm-kv-checksum-coverage-per-request.txt", "jsonld": "https://wpnews.pro/news/show-hn-sdcguard-inspect-vllm-kv-checksum-coverage-per-request.jsonld"}}