{"slug": "show-hn-external-kv-cache-offloading-cuts-long-horizon-inference-costs-by-50", "title": "Show HN: External KV Cache Offloading Cuts Long Horizon Inference Costs by 50%", "summary": "OpenLake, an open-source storage engine for offloading LLM KV caches from GPU memory to RAM and NVMe, cuts GPU time by 48.2% for long-context inference, reducing a 1,169-second workload to 606 seconds. The project achieves 1.72× lossless KV compression and 66× faster time-to-first-token at 128K context, dropping from 44 seconds to 0.6 seconds. OpenLake provides connectors for vLLM and SGLang without modifying inference engines.", "body_md": "Hey HN, we’re the developers of OpenLake, an open source storage engine for offloading LLM KV caches from GPU memory into a shared tier of RAM and NVMe. We built OpenLake because KV caches are outgrowing GPU memory.\n\nA single 256K token conversation on Gemma 4 31B produces approximately 43GB of KV state, more than half the memory of an 80GB H100. The problem becomes even harder across a cluster: a prefix cached on one GPU host is unavailable when the next request lands on a different GPU, forcing the new GPU to repeat work the fleet has already completed.\n\nOnce the KV cache is offloaded, network bandwidth becomes a major constraint on read latency. To move less data across the wire, we built deferred materialization: a custom CUDA kernel that losslessly compresses KV blocks before they leave GPU memory and decompresses them on the GPU after retrieval. In our tests, this achieved:\n\n- 1.72× lossless KV compression. - Approximately 600GB/s decompression throughput on an H100 - 80GB/s of effective KV throughput over a physical 50GB/s link\n\nAt 128K context, retrieving cached KV reduces TTFT from 44 seconds to 0.6 seconds, a 66× improvement. Across the complete workload, GPU time reduces from 1,169 seconds to 606 seconds, saving 48.2% of GPU cost.\n\nOpenLake is written in Rust and uses io_uring with one pinned runtime per physical core. We provide connectors for vLLM and SGLang so the cache can be enabled without modifying the inference engine itself.\n\nI would love to hear how others are handling KV reuse across GPU hosts, especially for long contexts, and get to know your thoughts.\n\nThanks!\n\nGitHub: [https://github.com/openlake-project/openlake](https://github.com/openlake-project/openlake)\n\nHere is our blog: [https://cloud.theopenlake.com/blog/taming-the-beast-managing...](https://cloud.theopenlake.com/blog/taming-the-beast-managing-100-tb-of-kv-cache-on-open-source-inference)\n\nComments URL: [https://news.ycombinator.com/item?id=49057767](https://news.ycombinator.com/item?id=49057767)\n\nPoints: 10\n\n# Comments: 0", "url": "https://wpnews.pro/news/show-hn-external-kv-cache-offloading-cuts-long-horizon-inference-costs-by-50", "canonical_source": "https://github.com/openlake-project/openlake", "published_at": "2026-07-26 13:02:21+00:00", "updated_at": "2026-07-26 13:22:21.219168+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["OpenLake", "Gemma 4 31B", "H100", "vLLM", "SGLang", "Rust", "io_uring"], "alternates": {"html": "https://wpnews.pro/news/show-hn-external-kv-cache-offloading-cuts-long-horizon-inference-costs-by-50", "markdown": "https://wpnews.pro/news/show-hn-external-kv-cache-offloading-cuts-long-horizon-inference-costs-by-50.md", "text": "https://wpnews.pro/news/show-hn-external-kv-cache-offloading-cuts-long-horizon-inference-costs-by-50.txt", "jsonld": "https://wpnews.pro/news/show-hn-external-kv-cache-offloading-cuts-long-horizon-inference-costs-by-50.jsonld"}}