Nvidia unveils alternative high bandwidth technology to bolster AI cards

Nvidia introduced NVHBM, a custom High Bandwidth Memory architecture designed to deliver 30% more memory bandwidth, 15% lower power consumption, and more usable silicon area than conventional HBM4E, targeting hyperscalers and AI companies building custom accelerators. Announced alongside its quarterly earnings, NVHBM is part of Nvidia's NVLink Fusion platform, with Amazon's Annapurna Labs as the first partner. Nvidia claims the combined architectural changes can yield approximately a 30% improvement in end-to-end XPU performance.

Nvidia is once again making its own parts for AI rather than relying on the rest of the industry to do it. In this case, it has introduced a customized High Bandwidth Memory architecture, NVHBM, that the company says can deliver significantly more bandwidth, lower power consumption and offer more usable silicon area than conventional HBM4E. To be sure, Nvidia will not be making the memory. It doesn’t make its own chips and has no foundry, after all. One of the big three memory makers – Micron, SK Hynix, or Samsung — we’ll actually make the chips. Nvidia is just designing them. The new memory was announced in a blog post https://blogs.nvidia.com/blog/nvlink-fusion-nvhbm-custom-high-bandwidth-memory on the same day as the company’s quarterly earnings call. It is positioned as an expansion of Nvidia’s NVLink Fusion https://www.nvidia.com/en-us/data-center/nvlink-fusion/ platform. NVHBM is aimed directly at hyperscalers and AI companies developing their own custom accelerators, or XPUs. Amazon’s Annapurna Labs https://amazon.jobs/content/en/teams/annapurna-labs will be the first announced partner to work with Nvidia on the technology. HBM that has emerged as a significant technology in the AI hardware space. Rather than putting memory in DRAM sticks that sit on the motherboard like we do now, HBM chips sit right next to the accelerator die. This greatly reduces latency and travel time for the day that moving between memory and accelerator. There is a significant difference between NVHBM and regular HBM memory: Nvidia is moving the memory controller off the XPU and into the HBM stack’s base die. In conventional HBM implementations, the memory controller resides on the accelerator’s main compute die. That requires substantial silicon area and I/O connections between the processor and HBM. Nvidia’s approach uses a custom memory controller and physical interface, or PHY, into the HBM base die. Nvidia says this reduces the amount of interface circuitry required on the XPU and creates more room for compute resources. Nvidia is never shy about bragging and it’s not about to start with NVHBM. It claims the memory can provide: Those gains are potentially more important than simply increasing raw memory bandwidth. Freeing silicon area means accelerator designers can add more compute engines, cache, specialized AI functions or other workload-specific circuitry without increasing the package footprint. NVHBM is not being introduced as a standalone memory product. Nvidia is positioning it as another component of NVLink Fusion, its effort to allow hyperscalers and AI companies to build custom silicon while plugging those processors into Nvidia’s broader rack-scale infrastructure. “Nvidia is establishing a standard NVHBM implementation, available from multiple memory providers. This reduces the engineering effort required to integrate and qualify memory across multiple suppliers — giving NVLink Fusion customers a faster path for bringing custom AI chips to market,” the company wrote in the blog post. Nvidia says the combined architectural changes can produce approximately a 30% improvement in end-to-end XPU performance, when the 30% memory-bandwidth improvement, additional compute-die area and 15% reduction in HBM power are considered together.