Photo: Merlin Lightpainting / Pexels
The joint system stores AI models' working memory on shared flash, and benchmarks show up to 56x faster first responses on long prompts
Super Micro Computer, IBM and Nvidia have teamed up on a fix for one of AI’s more expensive habits: doing the same math twice.
On November 5, 2026, the three companies announced a joint key-value (KV) cache platform built for large-scale AI inference. The goal is to stop GPUs from recomputing context they have already processed. Benchmarks from a validated reference architecture show time-to-first-token speedups of up to 56x on contexts reaching 130,000 tokens.
What the three companies actually built #
When a model reads a prompt, it builds an internal record of that context, known as the KV cache. If the cache gets evicted from memory, the GPU has to rebuild it from scratch on the next turn. That rebuilding is called recomputation, and it burns expensive GPU cycles.
The new platform is designed to keep that record around. It combines four pieces of hardware and software:
Supermicro’s Petascale storage servers provide the physical storage layer.
IBM Storage Scale software manages high-performance shared storage across the system.
Nvidia Dynamo handles the inference software side.
AI, tech, and the markets they move—in one daily briefing.
Daily. Free. Join 34,000+ readers across crypto, finance, and policy.
Nvidia Spectrum-X Ethernet switches move data between the components.
The target workloads are the long-context ones. That includes retrieval-augmented generation (RAG), where a model pulls in outside documents to answer a question, and multi-turn agents, which carry on extended back-and-forth tasks.
The benchmark numbers #
The performance figures come from a validated reference architecture compared against conventional GPU-recompute baselines.
Time-to-first-token, or TTFT, measures how long a user waits before the model starts responding. Under the new setup, TTFT stayed nearly flat as prompt sizes grew. At contexts of up to 130,000 tokens, the speedup reached up to 56x.
Throughput told a similar story. Concurrent requests per second (RPS) climbed from 0.19 to 4.26, a 22x improvement. Total processing time for a batch of 200 requests dropped by 95%.
The companies also ran what is known as a noisy-neighbor test, simulating a production environment where other traffic competes for the same resources. Even under that competing load, the system sustained 3.6 RPS, an 18x improvement. It completed 200 requests in 55.56 seconds during the noisy tests.
Part of Nvidia’s bigger memory push #
The KV-cache platform falls under Nvidia’s Inference Context Memory Storage Platform (ICMSP) initiative, which Nvidia introduced at CES 2026. The collaboration between IBM, Nvidia and Supermicro also predates this announcement, with the three having worked on related efforts since mid-2025.
The KV cache gets spread across several layers of memory: GPU high-bandwidth memory (HBM), DRAM, SSDs, and shared flash storage. Hot data stays close to the processor. Less urgent context moves further out, but remains reachable without a full rebuild.
Disclosure: This article was edited by Diego Almada Lopez. For more information on how we create and review content, see our Editorial Policy.