cd /news/artificial-intelligence/shadowserve-efficient-kv-cache-fetch… · home › topics › artificial-intelligence › article
[ARTICLE · art-170238] src=systems.seas.harvard.edu ↗ pub= topic=artificial-intelligence verified=true sentiment=↑ positive

ShadowServe: Efficient KV Cache Fetching with SmartNICs

Harvard University Ph.D. student Xingyu Xiang presented ShadowServe, an inference system that offloads KV cache fetching and hardware-accelerated decompression to the front-end SmartNIC, improving peak inference throughput by up to 1.42x over GPU, GPU codec, CPU, and uncompressed KV fetching baselines. ShadowServe was implemented on NVIDIA BlueField-3 and integrated with vLLM, using a parallelized run-to-completion data path, cache-resident data exchange, and pre-allocation memory management to work within the SmartNIC's limited resources. The approach targets front-end network bandwidth bottlenecks in KV cache fetching without interfering with host GPU or CPU computation.

read1 min views1 publishedNov 16, 2026

Abstract #

KV cache fetching is increasingly bottlenecked by front-end network bandwidth. Compression mitigates this issue, but the received KV cache must be decompressed before use. Existing approaches decompress KV caches using GPUs, GPU video codecs, or host CPUs. However, GPU kernels interfere with model computation, while video codecs and CPUs have limited decompression throughput. We present ShadowServe, an inference system that offloads KV cache fetching to the front-end SmartNIC. This enables high-throughput fetching through hardware-accelerated decompression, without interfering with the host GPU or CPU. However, efficiently off the heterogeneous KV fetching pipeline requires careful use of the SmartNIC’s limited resources. We address this with a parallelized run-to-completion data path, cache-resident data exchange, and pre-allocation memory management. We implement ShadowServe on NVIDIA BlueField-3 and integrate it with vLLM. ShadowServe consistently outperforms GPU, GPU codec, CPU, and uncompressed KV fetching baselines, improving peak inference throughput by up to 1.42x.

Bio #

Xingyu Xiang is a Ph.D. student in Computer Science at Harvard University, advised by Minlan Yu and Eddie Kohler. His research focuses on systems and networking for large language model inference. He received his bachelor’s degree in Computer Science from Peking University, and has previously published work at NSDI and SIGCOMM.

── more in #artificial-intelligence 4 stories · sorted by recency
── more on @shadowserve 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/shadowserve-efficien…] indexed:0 read:1min 2026-11-16 · —