Supermicro announced a KV cache storage solution with IBM and NVIDIA today that moves the key-value state behind long-context inference off the GPU and onto a shared all-flash tier. The package combines Supermicro Petascale storage servers on AMD EPYC 9005 processors, IBM Storage Scale, NVIDIA Dynamo, and NVIDIA Spectrum-X Ethernet, and the companies back it with a validated reference architecture that IBM published as a Redbook. The headline figure is a 56x cut in time to first token on 130,000-token prompts against a GPU that recomputes the context from scratch.
Why KV Cache Needs a Storage Tier #
Inference runs in two phases: prefill turns the prompt into tokens and pushes them through the model to produce the intermediate key-value states, and decode generates the response one token at a time against those states. Keeping the KV state in a persistent cache lets the system reuse it for the next turn, the next agent step, or the next request that shares the same context, and skip the recompute.
Agentic workflows, coding assistants, long-context RAG, and multi-turn applications generate enough of this state to overflow GPU and CPU memory quickly, which is why Supermicro, IBM, and NVIDIA are making a network storage tier.
In this design, NVIDIA Dynamo manages the cache hierarchy and moves blocks over NIXL; IBM Storage Scale Erasure Code Edition provides the shared file tier; and the Petascale servers supply the flash and the PCIe Gen5 lanes behind it. Supermicro says the three companies tested the stack at cluster level under high concurrency and with noisy-neighbor network traffic, and it is pitching the result as a turnkey build with factory rack integration and on-site deployment that it says cuts deployment time from weeks to days.
The Tested Configuration #
The reference architecture uses eight Supermicro ASG-1115S-NE316R Petascale nodes, each a 1U system with a single AMD EPYC 9535 and 16 Micron 7.68TB E3.S NVMe drives, which puts 122.88TB of raw flash in each node and about 983TB across the eight. Each node carries a dual-port 200Gb NVIDIA BlueField-3 B3220 on the client side and a 400Gb ConnectX-7 for the storage backend, with three Spectrum-X SN5600 switches in a spine and leaf layout and 800Gb/s uplinks.
The storage nodes run RHEL 9.6 and IBM Storage Scale ECE 6.0.0.1 in an 8+2P erasure code layout, and IBM puts the read throughput of this eight-node tier at 315GB/s from earlier testing.
| Component | KV Cache Reference Architecture (as tested) |
|---|---|
| Storage Tier | |
| Storage nodes | 8x Supermicro ASG-1115S-NE316R (1U, single socket) |
| CPU per node | AMD EPYC 9535, 128 lanes of PCIe Gen5 (64 to NVMe, 64 to networking) |
| Flash per node | 16x Micron 7.68TB E3.S NVMe (122.88TB raw) |
| Network per node | NVIDIA BlueField-3 B3220, dual-port 200Gb (client side) NVIDIA ConnectX-7 400Gb (storage backend) |
| Switching | 3x NVIDIA Spectrum-X SN5600, spine and leaf, 800Gb/s uplinks |
| Storage software | IBM Storage Scale Erasure Code Edition 6.0.0.1, 8+2P RHEL 9.6 |
| Inference Client | |
| System | Supermicro SYS-212GB-FNR (NVIDIA MGX), Ubuntu 24.04 |
| GPU used for the benchmarks | 1x NVIDIA RTX PRO 6000 Blackwell, 19.6GiB left for KV cache after weights and CUDA graphs |
| Client networking | 2x NVIDIA BlueField-3 B3140H 400Gb/s, up to 80GB/s client-facing bandwidth |
| Model and software | openai/gpt-oss-120b (MXFP4) vLLM 0.14.1 NVIDIA Dynamo 0.9.0 with NIXL NVIDIA AIPerf benchmark client |
| Cache tiering | Host and local tiers set to zero; all 1.4 million cached tokens served from the Storage Scale tier |
The benchmark client is a single Supermicro SYS-212GB-FNR MGX system, and results come from one NVIDIA RTX PRO 6000 Blackwell GPU in that box running the openai/gpt-oss-120b model in MXFP4 under vLLM 0.14.1 and Dynamo 0.9.0.
After the weights and CUDA graphs, 19.6GiB of GPU memory was left for KV cache, enough for about 1.14 million tokens on the GPU itself. For the tests, the host memory and local tiers were set to zero so that every cached block came back from Storage Scale, so the results measure the storage path by itself, with no help from a mixed hierarchy.
56x on Time to First Token, 22x on Throughput #
With no cache, time to first token for the 120B model rose from 0.572 seconds at 10,000 input tokens to 32.14 seconds at 130,000 as the GPU recomputed the full context. With the KV cache on Storage Scale, TTFT stayed between 0.193 and 0.57 seconds across the same range, which is the 56x figure at the 130,000-token mark and the 0.377-second spread Supermicro cites in its announcement. IBM, Supermicro, and NVIDIA ran these tests, and we did not run or audit the results.
The throughput test sent 200 requests over 28 concurrent connections, drawn from 100 unique prompts totaling 24 million tokens and 825GB of KV cache data, with the cache starting empty. The recompute baseline finished at 0.19 requests per second in 1,048 seconds; the Storage Scale tier finished at 4.26 requests per second in 46.94 seconds, a 22x gain that includes the cold-start cost of filling the cache.
A third run added four clients pushing 200GB/s of unrelated network traffic across the same fabric, and the cached configuration held 3.6 requests per second and 55.56 seconds for the 200 requests, about nine seconds slower than the clean run and still 18x the baseline. IBM attributes the small penalty to the Spectrum-X fabric isolating the cache traffic from the background load.
“Agentic AI and long-context applications depend on reusing context efficiently across many steps, and NVIDIA Dynamo helps orchestrate distributed inference so enterprises can get more usable throughput from every GPU,” said Shruti Koparkar, NVIDIA AI Inference Product Lead. “By pairing NVIDIA Dynamo and Spectrum-X Ethernet with Supermicro Petascale storage servers and IBM Storage Scale, this KV cache solution creates a high-performance shared storage tier that can reduce redundant computation, improve time-to-first-token and help customers scale resilient AI inference services.”
Sizing and What Is Not in the Numbers #
IBM’s sizing guide puts a four-node build at 4+2P with 67 percent usable capacity for 7B to 34B models and up to 64 GPUs, the tested eight-node build at 8+2P and 80 percent usable for 70B to 120B models and 100 to 256 GPUs needing 100 to 300GB/s of read bandwidth, and 16 or more nodes at 16+3P and 84 percent usable for larger models and 300 or more GPUs at 300GB/s and up. Storage Scale ECE itself scales from three storage servers to 256.
The single-GPU test setup is the main caveat. The reference architecture is sized for hundreds of GPUs, but every result in the paper comes from one RTX PRO 6000 against an eight-node, 983TB flash tier that IBM rates at 315GB/s, so the storage side was nowhere near saturation, and the tests do not show how TTFT holds up when a full GPU cluster is pulling cache at once.
Supermicro has not published pricing or an availability date for the packaged solution, and the announcement gives no GPU utilization figures. The approach lands in the same territory as DDN’s Dynamo-integrated KV cache acceleration on the AI400X3M, with the difference that this stack runs on commodity Supermicro servers and Ethernet under IBM’s file system.
“As AI workloads expand from training to inference, enterprises and neoclouds deploying high-throughput inference workloads characteristic of agentic AI and long-context, multi-turn generative AI need to remove bottlenecks to improve GPU efficiency,” said Charles Liang, president and CEO of Supermicro. “Our solution with IBM and NVIDIA clearly demonstrates these performance benefits.”