{"slug": "breaking-the-kv-cache-memory-wall-using-dynamo-and-liqid-cxl-shared-memory", "title": "Breaking the KV cache memory wall using Dynamo and Liqid CXL shared memory", "summary": "A Micron team led by Dr. Skyler Windh built cross-host KV cache sharing into Nvidia Dynamo and ran it on the Great Abaco cluster, a CXL 2.0 shared-memory system designed with Liqid for the US Department of Energy's Pacific Northwest National Laboratory, reporting that a 32 GB CXL cache per server delivered 2.8× the throughput and 2.9× faster time to first token than DRAM of the same size on Llama3-70B with 90 percent shared prefixes. Hosts reusing each other's KV blocks got 3× the throughput with half the memory, and CXL fabric-attached memory ran within 99 percent of host DRAM. Liqid SVP for Solution Architecture Martin Cooper said the work shows shared CXL memory can lift GPU inference throughput 2.8× over DRAM, with no vLLM patches and no changes to the GPU servers.", "body_md": "ARCHITECTURE\n\n# Breaking the KV cache memory wall using Dynamo and Liqid CXL shared memory\n\nMicron has found that KV cache memory wall limitations on a GPU server can be substantially reduced by using a Liqid CXL-accessed memory pool to enlarge the DRAM cache.\n\nA Micron team is building a GPU cluster for the US Department of Energy’s Pacific Northwest National Laboratory (PNNL) to run next-generation, large-scale scientific, AI, HPC and data-intensive workloads. The cluster is called [Great Abaco](https://www.liqid.com/products/abaco), and Dr. Skyler Windh and his team at Micron developed a high-capacity shared memory system architecture for AI and HPC workloads. It uses CXL 2.0.\n\nThe system is described in a 60-minute webinar and the background is this: as Large Language Model (LLM) inference scales across multi-GPU deployments, the Key-Value (KV) Cache becomes a major performance bottleneck. Each GPU typically keeps its own isolated cache, which can spill to host DRAM. When different GPUs process the same prompt prefix, each one repeats the same prefill computation. CXL fabric-attached memory (FAM) addresses this by providing multiple GPUs with a shared, byte-addressable memory pool, allowing other GPUs to reuse KV Cache entries computed by one GPU.\n\nWindh and his team built cross-host KV sharing into Nvida Dynamo and ran it on Micron’s Great Abaco cluster.\n\n [Abaco](https://www.liqid.com/products/abaco) is a shared-memory system architecture for AI and HPC, designed by Micron with Liqid as the memory fabric and orchestration partner. A single Abaco rack presents up to 160TB of DDR5 as an addressable pool that one or multiple hosts in the rack can claim, share, or release without a reboot and without re-cabling. Applications reach the pool through a standard file system interface, so most code runs unmodified.\n\nThe cluster uses H200 NVL GPUs in a Liqid PCIe Gen5 chassis, with four Liqid CXL memory chassis forming one shared pool.\n\nOn Llama3-70B with 90 percent shared prefixes, a 32 GB cache per server delivered 2.8× the throughput and 2.9× faster time to first token than DRAM of the same size. Hosts reusing each other’s KV blocks got 3× the throughput with half the memory, and CXL memory ran within 99 percent of host DRAM.\n\nIe. CXL FAM performs on par with a host DRAM baseline.\n\nThe Micron team integrated CXL FAM-based KV Cache sharing into an open-source multi-GPU inference framework, with:\n\n- A block-level cache manager that allocates in CXL memory\n- Sequence-hash-based cache discovery across GPUs\n- An asynchronous transfer engine that moves blocks between CXL and GPU memory without stalling the inference scheduler\n- A CUDA host-registration workflow that enabled direct memory access (DMA) between GPUs and CXL memory\n\nLiqid SVP for Solution Architecture Martin Cooper says this Micron work shows that shared CXL memory can lift GPU inference throughput 2.8× over DRAM. Liqid puts GPUs and CXL memory on one PCIe fabric, so memory scales separately from compute. The serving stack stays as it is: no vLLM patches and no changes to the GPU servers. A shared pool lets you right-size each server and buy the memory once.\n\nYou can access the webinar 's 30-slide presentation deck [here](https://computeexpresslink.org/wp-content/uploads/2026/10/CXL_Consortium_KV_Cache_Sharing_FINAL.pdf), and watch the whole webinar [here](https://www.youtube.com/watch?v=cGTqLDHrcgs) - it's has many YouTube adverts though. You can download a transcript we created [here](<https://www.blocksandfiles.com/files/2026/10/23/Micron CXL webinar transcript.pdf>) if you wish, and read it along with the presentation deck.\n\nDownload an Abaco technical brief [here.](https://cdi.liqid.com/hubfs/LIQID%20Abaco%20TB_080326.pdf?_gl=1*1c4sgr1*_gcl_au*NTc0MTUxNzI3LjE3OTI3Njk2NTg.*_ga*ODYxNTc1NzIuMTc5Mjc2OTY1OA..*_ga_DWS4FKCM70*czE3OTI3Njk2NTgkbzEkZzAkdDE3OTI3Njk2NTgkajYwJGwwJGgw*_ga_ZRS8LKLNDK*czE3OTI3Njk2NTgkbzEkZzAkdDE3OTI3Njk2NTgkajYwJGwwJGgw)\n\n##### Bootnote\n\nGFAM, referred to in the slides = Global Fabric Memory", "url": "https://wpnews.pro/news/breaking-the-kv-cache-memory-wall-using-dynamo-and-liqid-cxl-shared-memory", "canonical_source": "https://www.blocksandfiles.com/architecture/2026/10/26/breaking-the-kv-cache-memory-wall-using-dynamo-and-liqid-cxl-shared-memory/5305130", "published_at": "2026-10-26 09:38:16+00:00", "updated_at": "2026-10-26 10:43:19.133163+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-chips", "mlops"], "entities": ["Micron", "Liqid", "Nvidia", "Dynamo", "Great Abaco", "Pacific Northwest National Laboratory", "Skyler Windh", "Martin Cooper"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/breaking-the-kv-cache-memory-wall-using-dynamo-and-liqid-cxl-shared-memory", "markdown": "https://wpnews.pro/news/breaking-the-kv-cache-memory-wall-using-dynamo-and-liqid-cxl-shared-memory.md", "text": "https://wpnews.pro/news/breaking-the-kv-cache-memory-wall-using-dynamo-and-liqid-cxl-shared-memory.txt", "jsonld": "https://wpnews.pro/news/breaking-the-kv-cache-memory-wall-using-dynamo-and-liqid-cxl-shared-memory.jsonld"}}