UlaCache: Unified Logical Addressing for Sparse LLM Serving

A paper submitted to arXiv on 21 Oct 2026 presents UlaCache, a KV cache management system that uses a unified logical address space spanning GPU KV pools and optional CPU memory to serve sparse-attention LLM inference. With speculative decoding enabled under matched CPU and GPU memory budgets, UlaCache reached peak decoding throughput speedups of 1.16x (data-parallel) and 1.70x (tensor-parallel) on H200, and 1.73x (DP) and 2.51x (TP) on B300 versus the state-of-the-art baseline. UlaCache's Parallel Bucketed LRU (PB-LRU) updates recency metadata and selects eviction victims in parallel, and local GPU buffers retain selected KV across decoding steps so only missing entries require transfers.

Computer Science Distributed, Parallel, and Cluster Computing Submitted on 21 Oct 2026 Title:UlaCache: Unified Logical Addressing for Sparse LLM Serving View PDF https://arxiv.org/pdf/2610.29820 HTML experimental https://arxiv.org/html/2610.29820v1 Abstract:Long contexts increase the memory cost of the key-value KV cache in large language model LLM inference. Sparse attention reduces attention computation and memory traffic by selecting historical tokens, but the retained KV cache must remain available for later decoding steps. We present UlaCache, a KV cache management system built around a unified logical address space spanning GPU KV pools and optional CPU memory. Stable logical references let the inference engine allocate and select KV independently of physical placement. Common allocation and sparse-access interfaces support data-parallel DP and tensor-parallel TP attention. For distributed storage, local GPU buffers retain selected KV across decoding steps, so only missing entries require transfers. When CPU storage is enabled, Parallel Bucketed LRU PB-LRU manages a GPU pool shared across requests. It updates recency metadata and selects eviction victims in parallel. UlaCache transfers KV and then updates the address mappings. With speculative decoding enabled under matched CPU and GPU memory budgets, UlaCache achieves peak decoding throughput speedups of 1.16 $\times$ DP and 1.70 $\times$ TP on H200, and 1.73 $\times$ DP and 2.51 $\times$ TP on B300, compared with the state-of-the-art baseline. Additional Features References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .