DapuStor's KV Cache NVMe SSD offload explainer

DapuStor published a 5-page explainer document showing how tiering evicted KV cache contents to NVMe SSDs can solve the recomputation delay problem in LLM inference, which is limited first by GPU HBM capacity and then by tiered caching through x86 host DRAM to internal and external NVMe SSDs. The document describes how vLLM and LMCache pack KV cache data from multiple tokens for transfer, letting external storage generate flash-friendly sequential large-block I/O patterns that improve bandwidth utilization on NVMe SSDs, illustrated with diagrams and Python code examples. DapuStor notes PCIe Gen5 NVMe SSD capacities reaching 245 TB, with more expected next year, leaving scope for petabyte-level external KV cache storage.

DapuStor's KV Cache NVMe SSD offload explainer AI inferencing is memory-limited, bottlenecked by KV Cache https://www.blocksandfiles.com/ai-ml/2026/03/30/nvidia-and-its-partners-kv-cache-extenders/5209284 capacity limits dictated by a GPU's high-bandwidth memory HBM amount and then by the tiered KV caching scheme via the x86 hosts DRAM, through to internal and external NVMe SSDs. We have discussed KV cache news and developments many times and DapuStor https://www.blocksandfiles.com/flash/2026/09/21/a-look-at-dapustors-combined-slc-and-qlc-ssd/5297708 has put together a 5-page explainer document that shows how tiering evicted KV cache contents to NVMe SSDs can solve the recomputation delay problem. The document says offloading large KV Cache datasets to a more cost-effective external storage tier is emerging as a practical approach for scaling inference clusters. It neatly explains how KV Cache works in LLM inference and discusses how NVMe SSDs can help build a cost-efficient storage layer for AI infrastructure. The document discusses the mechanisms used by vLLM https://github.com/vllm-project/vllm and LMCache https://lmcache.ai/en/ to make SSD-based KV Cache more effective. These are explained with text, diagrams and a couple of Python code examples. For example; It says "Both vLLM and LMCache support packing KV Cache data from multiple tokens for transfer, improving memory efficiency and I/O efficiency." The point here is that "External storage systems can use these larger chunks to generate flash-friendly, sequential large-block I/O patterns, improving bandwidth utilization on NVMe SSDs." With PCIe gen5 NVMe SSDs having capacities rise to 245 TB, and hopefully beyond that next year, there is a lot of scope for building petabyte-level external storage KV caches with good bandwidth and high capacity. Our version of DapuStor's paper, generalized to all NVMe SSDs, can be downloaded here.