CEA-Leti Pushes Stacking Roadmap as AI Runs Into Memory and Power Limits

CEA-Leti program manager Pascal Vivet said 3D stacking and chiplet-based architectures are key to overcoming AI's memory and power density limits, as the institute develops interconnects from 10-micron-class to below 1 micron, including a 1-micron pitch die-to-wafer hybrid-bonding demonstration and a 200 nm wafer-to-wafer target. Vivet emphasized moving from HBM to slower, wider, closer memory interfaces to support future systems with hundreds of gigabytes to terabytes of memory near compute, while noting power density, not power budget, is the main limitation.

As AI models grow, hardware designers face a new challenge: placing far more memory closer to processors, moving data with less energy, and cooling systems whose power density keeps rising. For Pascal Vivet, program manager for advanced programs at CEA-Leti, that challenge is pushing 3D integration and chiplet-based architectures from advanced packaging technologies into the center of AI system design. “3D technology is key,” Vivet told EE Times, because it allows the industry to scale “horizontally, vertically, and to assemble the best technology for chiplet partitioning in an open ecosystem.” From high bandwidth to wide bandwidth The immediate driver is the scale of AI models and the memory capacity required to support them. Vivet said the main challenge is “clearly the size of the computing models and the size of memory.” The goal, he said, is to push available technologies far enough to be ready within the next few years for systems with very large amounts of memory placed as close as possible to the compute engine. He referred to memory capacities in the hundreds of gigabytes and possibly into the terabyte range integrated close to the chip. View All https://www.eetimes.com/category/sponsored-content/ “For that, 3D stacking will be key,” Vivet said. High-bandwidth memory has already become central to AI accelerators, but Vivet doesn’t see today’s HBM architecture as the end point. HBM increases bandwidth by placing stacked DRAM beside processors, but the connection remains side-by-side. That limits how wide and energy-efficient the interface can become. “HBM means high bandwidth, but it’s not wide enough,” Vivet said. The direction Leti is exploring isn’t simply more bandwidth in the conventional sense. It’s a move toward physically closer, more parallel memory interfaces. Instead of relying only on faster signaling links between adjacent dies, future systems could stack memory directly above the compute engine or place it nearby using denser, lower-energy interfaces. “We want to move from HBM to something slower, wider, closer, using the right technologies,” Vivet said. This is especially important for inference, where much of the workload involves repeatedly reading model weights. Dense memories optimized for read energy could therefore play a major role if they can be integrated close enough to compute. Vivet’s focus is where such memories sit, how they connect to compute, and whether the resulting system can be powered, cooled, tested, and manufactured. Leti is developing a broad 3D integration toolbox. Vivet said the institute is working on technologies ranging from 10-micron-class interconnects to below 1 micron, including die-to-wafer and wafer-to-wafer hybrid bonding, ultra-dense through-silicon vias, fan-out wafer-level packaging, and chiplet integration. He cited a recent demonstration of die-to-wafer hybrid-bonding interconnect at 1-micron pitch and a wafer-to-wafer hybrid-bonding target of 200 nm. The point isn’t that every application will use the most aggressive option. Data-center AI systems may require very dense memory-on-compute integration. Automotive, space, and defense applications may benefit first from chiplets and advanced packaging that provide access to advanced functions without requiring every block to be built on the same leading-edge die. Stacking creates a power-density problem Stacking memory and logic closer together can reduce data-movement energy, but it also creates a harder thermal problem. “The main limitation is not the power budget itself, but the power density,” Vivet said. That distinction matters because 3D integration increases functional density. It can shorten interconnects and improve bandwidth, but it also concentrates heat. Existing cooling technologies define the maximum power density a system can tolerate. Vivet used water cooling as an example. In advanced computing farms, water cooling can extract more heat than conventional air cooling, but it still defines a ceiling. For a given area, the cooling method determines how much thermal energy can be removed, which in turn limits how much power can be dissipated in the chip. “Water cooling defines the maximum power density of the chip,” Vivet said. That makes architectural choices inseparable from thermal choices. A data center accelerator may be able to rely on liquid cooling, while an edge AI system in a car, aircraft, drone, or defense platform faces a very different thermal envelope. In each case, the integration scheme must match the application’s cooling capacity. This is one reason Vivet emphasizes wider and slower interfaces. More parallel communication at lower speed can help reduce energy per bit and ease the power-density problem. The same logic applies to electrical interconnects https://www.edn.com/optimize-hi-rel-designs-with-rugged-flexible-interconnects/ and, in some cases, photonics. Power delivery also must be reconsidered. As compute and memory are stacked more tightly, designers must decide how power reaches the compute engine and where power management functions should sit. “Power management needs to be re-architected very early,” Vivet said. Backside power delivery may be a first step, but Vivet said the roadmap must go further. Future systems may require point-of-load power delivery, integrated passives, and power-management functions co-integrated close to the compute engine, potentially as backside or chiplet-based functions. The challenge isn’t only technical. It’s architectural. If power delivery, cooling, and memory placement are considered too late, the system may be impossible to optimize. The package can no longer be treated as a passive container for completed chips. Chiplets need early system co-optimization That shift leads to what Vivet sees as one of the hardest parts of the roadmap. “The most difficult is early partitioning of the system in terms of technology layers and functions,” he said. In conventional design flows, packaging decisions often come after circuit and architecture decisions. In advanced AI systems, that sequence no longer works. Memory capacity, interconnect density, power budget, thermal budget, cooling technology, cost, and yield must be considered at the beginning. In practice, that means estimating power, thermal behavior, cooling limits, integration scheme, and cost before the physical architecture is locked. Vivet said Leti is setting up methodologies for early analysis of power and thermal budgets, using knowledge of the cooling technology and 3D integration scheme before the full system is designed. Cost modeling is also essential, because the most advanced stacking option may not be appropriate for every application. Virtual prototyping will become increasingly important. Vivet said future architectures need early software models annotated with technology information such as density, power, and thermal behavior. Those models could help system designers explore where to place functions before committing to physical implementation. This is also where chiplet ecosystems become critical. For Europe, chiplets matter because they could allow companies to share IP, access silicon, and combine advanced functions, packaging technologies, and application-specific accelerators without putting every block on the same leading-edge die. For an open chiplet model to scale, the industry needs more than physical integration technology. Vivet pointed to die-to-die communication standards such as UCIe, design-for-test standards, test-and-repair methods, and chiplet design kits that allow chiplets to be treated more like reusable IP blocks. For Vivet, the future of AI hardware won’t be determined only by the fastest processor or the densest memory. It will depend on how intelligently the industry stacks, connects, powers, and cools them. The result is a new role for advanced packaging. It’s no longer simply the last step in chip assembly. In AI systems, it’s becoming part of the architecture itself. Read also: From Co-Packaged Optics to Nanolasers, Photonics Moves Inward https://www.eetimes.com/from-co-packaged-optics-to-nanolasers-photonics-moves-inward/ Intel Foundry Improves Execution, but External Customers Remain the Test https://www.eetimes.com/intel-foundry-improves-execution-but-external-customers-remain-the-test/ CEA-Leti Looks Beyond SRAM and DRAM as AI Reshapes the Memory https://www.eetimes.com/cea-leti-looks-beyond-sram-and-dram-as-ai-reshapes-the-memory-roadmap/