Five Ways NVIDIA Co-Design Maximizes the Value of Every Byte in the AI Factory

NVIDIA says its Vera Rubin NVL72 delivers up to 30x the throughput per megawatt of GB300 NVL72 on SemiAnalysis AgentX at 160 tokens per second per user, even though per-GPU HBM capacity held at 288 GB while bandwidth rose to 19.2 TB/s. NVIDIA attributes the gain to co-design across GPU architecture, the NVLink scale-up interconnect, serving software, CPU and storage, including the NVFP4 format, which cuts model footprint roughly 3.5x versus FP16 and 1.8x versus FP8 and delivers up to 4x the throughput of an FP8 baseline at the same interactivity. NVFP4 KV cache cuts cache footprint by up to 50% versus FP8, keeping about 2x more context resident on device.

In an AI factory, high bandwidth memory HBM is the fastest, most expensive, and most contested memory tier. Every byte spent on a redundant weight copy, an idle session’s KV cache, or a recomputed prefix is a byte that is not being used to produce tokens. The question is no longer how much memory a GPU has, but how much useful work each byte delivers. The generational data in Figure 1 below shows why. From GB300 to NVIDIA Rubin, capacity held at 288 GB while bandwidth rose to 19.2 TB/s. agentic throughput per megawatt grew far faster. On SemiAnalysis AgentX https://developer.nvidia.com/blog/nvidia-vera-rubin-and-blackwell-set-a-new-standard-for-agentic-ai-performance-per-watt/ , which replays recorded coding-agent sessions with real tool-call gaps and growing context, NVIDIA Vera Rubin NVL72 then delivers up to 30x the throughput per megawatt of GB300 NVL72 at 160 tokens per second per user. Capacity alone cannot explain a 30x gain delivered from the same 288 GB per GPU. The difference is how much useful work each byte delivers. NVIDIA treats this as a co-design problem that spans GPU architecture, the scale-up interconnect, serving software, the CPU, and storage. This post examines five ways that co-design maximizes the value of each byte of memory, from the bits that encode a single weight to the shared storage that preserves and reuses context across agentic turns and nodes. 1. How does NVFP4 store more intelligence in fewer bytes? The most direct way to raise the value of a byte is to make each value smaller without losing what it encodes. NVFP4 https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/ does this with a 4-bit E2M1 element and two levels of scaling: Every 16-value micro-block shares an FP8 E4M3 scale factor, and a per-tensor FP32 scale sits on top. The result is a format that is small enough to free significant HBM, yet granular enough to preserve accuracy where naive 4-bit formats fall short. Applied to weights, NVFP4 reduces the model footprint by approximately 3.5x relative to FP16 and 1.8x relative to FP8, freeing memory for what agentic inference needs most: more KV-cache capacity, larger batches, and higher concurrency. NVFP4 has become the default precision for open agentic benchmarks with strict accuracy requirements, including AA-AgentPerf, SemiAnalysis AgentX, and MLPerf Endpoints. Across these benchmarks and leading mixture-of-experts MoE architectures, it delivers up to 4x the throughput of an FP8 baseline at the same interactivity Figure 3 below . The same principle extends to the KV cache itself. NVFP4 KV cache https://developer.nvidia.com/blog/optimizing-inference-for-long-context-and-large-batch-sizes-with-nvfp4-kv-cache/ cuts the cache footprint by up to 50% compared with FP8, so approximately 2x more context stays resident on device. Fewer evictions mean higher cache-hit rates during prefill, while a smaller cache means less bandwidth consumed during decode. NVFP4 shrinks the bytes. The next question is where those bytes live, and how many copies of them the AI factory has to keep. 2. How does the NVLink scale-up domain turn 72 GPUs into one memory pool? MoE models activate only a subset of parameters per token, but every expert’s weights must still live somewhere in HBM. When experts are packed onto a small number of GPUs, each GPU spends a large share of its memory on weights, and each decode step pays a heavy weight-loading cost for the MoE GroupGEMMs. The bytes are resident, but they are not working hard. Wide expert parallelism https://developer.nvidia.com/blog/scaling-large-moe-models-with-wide-expert-parallelism-on-nvl72-rack-scale-systems/ Wide-EP in NVIDIA TensorRT-LLM changes that ratio by spreading experts across eight or more GPUs. With DeepSeek-R1 at EP64, each GPU holds four experts per layer, or 232 expert blocks across 58 MoE layers out of 14,848 in the full model. Fewer expert weights per GPU means: - Capacity: More HBM per GPU is left for KV cache, larger batches, and higher concurrency. - Arithmetic intensity: GroupGEMM kernels reuse each loaded weight across more tokens, raising FLOPs per byte. - Balance: The Expert Parallel Load Balancer EPLB redistributes hot and cold experts between forward passes, so no GPU’s memory is overcommitted while another sits idle. This only works because the scale-up domain makes remote bytes nearly as useful as local ones. Every MoE layer triggers all-to-all dispatch and combine traffic, and without a high-bandwidth fabric, those collectives would erase the gains. On GB200 NVL72, 130 TB/s of aggregate NVLink bandwidth absorbs that traffic, and large EP EP32 delivers up to 1.8x more output tokens per second per GPU than small EP EP8 at 100 tokens per second per user. Vera Rubin NVL72 extends the same design point. Each Rubin GPU brings up to 288 GB of HBM4 and 3.0 TB/s of NVLink 6 bandwidth, for roughly 20.7 TB of HBM4 across the rack. Rubin also adds inline memory tensor acceleratory TMA descriptor updates, so one shared descriptor can serve every expert with a common layout, reducing the metadata overhead that grows with expert count. Wide-EP decides where weight bytes live. The next lever decides whether KV cache bytes, once computed, are ever computed again. 3. How does NVIDIA Dynamo make every cached byte count? The key-value KV cache stores intermediate attention calculations from previously processed tokens so the model can reuse them instead of recomputing them. This inference state is ephemeral and can be evicted and recomputed when needed, but KV cache is only valuable if it is reused. Agentic workloads are full of reuse opportunities: system prompts, tool schemas, shared documents, and the growing history of a multistep task. But if a request lands on a GPU that does not hold its prefix, or if that prefix was evicted moments earlier, the system pays for prefill again. The bytes existed; they were just in the wrong place. NVIDIA Dynamo https://developer.nvidia.com/blog/how-to-reduce-kv-cache-bottlenecks-with-nvidia-dynamo/ is the distributed serving layer that closes this gap. It treats KV cache as a fleet-wide resource rather than a per-GPU side effect, and it does so through three co-designed capabilities: - KV-aware routing: The Dynamo router tracks where KV blocks reside and considers both cached-prefix overlap and worker load when placing requests. It balances existing context against the cost of sending work to an utilized GPU and turns potential recomputations into cache hits. - Disaggregated serving: Prefill and decode run on separate GPU pools, each sized for its own compute and memory profile. Decode GPUs can dedicate HBM to KV cache and batch size rather than reserving headroom for bursty prefill. - KV Cache Runner KVCR : This tracks where every block lives across configured GPU, pinned CPU, and storage tiers, including local SSDs and mounted network filesystems. A prefix preserved outside HBM can be restored when a later request needs it, allowing useful context to survive GPU eviction and continue saving prefill work. KVCR uses the NVIDIA Inference Xfer Library NIXL https://github.com/ai-dynamo/nixl for its storage-transfer paths, allowing cached blocks to move while the GPU processes other work. KVCR also enables distributed reuse, with router-provided hints helping a selected worker retrieve cached context from a peer, while the inference engine retains control of GPU memory and local host offloading. The additional KVCR-Guard feature preserves committed KVCR-managed host-memory cache across an engine failure as long as the guard service and host remain available. Dynamo’s KV-cache offloading depends on fast retrieval: Preserving a prefix outside HBM saves time when loading it is faster than repeating prefill. NIXL connects this offloading path to external storage systems. In separate validation tests, storage vendors VAST Data and WEKA demonstrated 35 GB/s to one H100 GPU using GPUDirect Storage and up to 270 GB/s of aggregate read throughput across eight H100 GPUs through a custom NIXL plugin, respectively. These tests showed that external storage could supply KV cache to the tested GPU systems without becoming the transfer bottleneck, supporting cache capacity beyond HBM. A separate VAST test demonstrated the response-time benefit: Using vLLM and LMCache on a DGX H100 system, loading stored KV cache instead of recomputing it reduced time to first token for Qwen3-32B with a 130K-token prompt. Dynamo coordinates where cached bytes go and when they come back. The next two levers expand the capacity available for that context, starting with CPU memory. 4. How does the NVIDIA Vera CPU keep idle context from wasting HBM? Agentic inference does not run in one uninterrupted pass. A single session alternates between GPU model steps and CPU-side work: tool calls, code execution, retrieval, database queries, and sandboxed evaluation. During each of those gaps, any session KV cache retained in HBM occupies capacity without serving active token generation, while other requests compete for the same bytes. In a fully subscribed AI factory, the GPU does not wait. It takes on new requests while the tool call runs, and the longer the gap, the more likely the original session’s KV cache is evicted to make room. When the tool returns, the GPU may have to rebuild that context through a full prefill rather than resume from a cached state. A slow CPU therefore costs HBM twice: first as idle residency, then as recomputation. The NVIDIA Vera CPU https://developer.nvidia.com/blog/nvidia-vera-cpu-boosts-ai-factory-throughput-to-accelerate-agentic-workloads/ attacks both sides of this problem: - Shorter gaps: 88 Olympus cores deliver 1.8x faster single-threaded performance, 40% lower peak loaded latency than x86 CPUs, and up to 14 GB/s of memory bandwidth per core. Faster, more predictable tool calls shrink the window in which KV cache is exposed to eviction pressure. - A fast place to park context: Each Vera CPU provides up to 1.5 TB of LPDDR5X, more than 5x the HBM4 capacity of a single Rubin GPU. NVIDIA NVLink-C2C connects it coherently to Rubin at 1.8 TB/s, roughly 7x the 256 GB/s of an x16 PCIe Gen 6 link. Together, these capabilities change the economics of an idle session. Instead of holding its KV cache in HBM or discarding it, Dynamo’s KV Block Manager can stage it in Vera memory for the duration of the tool call and bring it back over NVLink-C2C when the agent resumes. HBM stays dedicated to sessions that are actively generating tokens, and the paused session avoids paying for prefill twice. Vera memory is the warm tier, sized for context that will return within a turn. For context that must survive across turns, agents, and nodes, the AI factory needs a tier that is larger still. 5. How does NVIDIA CMX extend context memory across the pod? Keeping every reusable KV-cache byte in GPU or host memory limits capacity for active inference, but discarding it can force recomputation.. Multi-turn sessions, long-horizon tasks, and multi-agent workflows that share working memory all generate KV cache that will be needed again, often minutes later and often on a different node. Longer context windows and more concurrent sessions increase KV-cache capacity requirements beyond GPU HBM G1 in Figure 9 below and host memory G2 . The traditional hierarchy was not designed for this data. Local SSDs G3 are bound to one node, while shared enterprise storage G4 is built for durable data and carries replication and metadata overhead that ephemeral KV cache does not need. NVIDIA CMX https://developer.nvidia.com/blog/introducing-nvidia-bluefield-4-powered-inference-context-memory-storage-platform-for-the-next-frontier-of-ai/ context memory storage closes this gap with a new G3.5 tier: Ethernet-attached flash, shared at the pod level, and optimized specifically for KV cache. It is built on the NVIDIA STX reference design and co-designed across four layers: - Storage processing: The NVIDIA BlueField-4 STX Storage Processor manages the NVMe SSDs, runs storage services, and offloads data integrity and encryption for KV cache. - Networking : NVIDIA Spectrum-X Ethernet provides low-latency, high-bandwidth RDMA access between compute nodes and CMX storage nodes. - KV-aware services: NVIDIA DOCA Memos provides KV-aware services and I/O control on BlueField-4, treating context as a first-class resource for placement and sharing. - Serving integration: Dynamo and NIXL make CMX appear as one seamless context tier across the pod, prestaging context and routing requests to where KV cache already resides. The pod-level design is what raises the value of each byte. A prefix computed once on one node can be reused across compatible model instances in the pod, so agents that share documents, tools, or working memory can avoid duplicate prefill. Pooling context storage across the pod reduces KV-cache duplication and stranded capacity across nodes, extending effective GPU memory for inference. By prestaging reusable context into GPU or host memory, CMX reduces inference stalls, while its KV-cache-optimized storage tier lowers the power overhead of serving that context. Together, these capabilities help CMX deliver up to 5x higher tokens per second and up to 5x better power efficiency compared with traditional storage for long-context inference. How do the five levers work together? Each lever removes a different kind of waste. NVFP4 removes bits a value does not need, Wide-EP removes redundant weight copies, Dynamo removes recomputation, Vera removes idle residency during tool calls, and CMX enables KV-cache sharing and reuse across nodes. Because these levers are designed as one memory system, not as independent parts, the gains stack: A 4-bit KV block saves space in HBM, in Vera memory, and in CMX alike. The result is an AI factory that measures memory not by capacity alone, but by the useful tokens each byte produces. Vera Rubin NVL72 shows what that looks like: up to 30x the throughput per megawatt of GB300 NVL72, from the same 288 GB per GPU. As agentic workloads grow, every byte counts. Learn more This work was made possible through the expertise and engineering contributions of Wei-ming Chen, Shiyang Chen, Chao Ni, Mengdi Wang, and Jeff Pool.