AI inferencing is memory-limited, bottlenecked by KV Cache capacity limits dictated by a GPU's high-bandwidth memory (HBM) amount and then by the tiered KV caching scheme via the x86 hosts DRAM, through to internal and external NVMe SSDs. We have discussed KV cache news and developments many times and DapuStor has put together a 5-page explainer document that shows how tiering evicted KV cache contents to NVMe SSDs can solve the recomputation delay problem.
The document says off large KV Cache datasets to a more cost-effective external storage tier is emerging as a practical approach for scaling inference clusters. It neatly explains how KV Cache works in LLM inference and discusses how NVMe SSDs can help build a cost-efficient storage layer for AI infrastructure.
The document discusses the mechanisms used by vLLM and LMCache to make SSD-based KV Cache more effective. These are explained with text, diagrams and a couple of Python code examples. For example;
It says "Both vLLM and LMCache support packing KV Cache data from multiple tokens for transfer, improving memory efficiency and I/O efficiency."
The point here is that "External storage systems can use these larger chunks to generate flash-friendly, sequential large-block I/O patterns, improving bandwidth utilization on NVMe SSDs."
With PCIe gen5 NVMe SSDs having capacities rise to 245 TB, and hopefully beyond that next year, there is a lot of scope for building petabyte-level external storage KV caches with good bandwidth and high capacity. Our version of DapuStor's paper, generalized to all NVMe SSDs, can be downloaded here.