DRAM is a lynchpin of AI memory, but it’s not the only critical component in the AI data center memory hierarchy.
Existing memories are getting reshuffled throughout the AI stack, balancing tradeoffs between capacity, latency, and power.
As high-bandwidth memory (HBM) gets gobbled up to meet AI demands, memory initially developed for smartphones and client devices, such as low-power DRAM (LPDDR) and small outline compression attached memory module (SOCAMM), are also finding a place in the stack as performance per watt becomes a key data center metric and inference workloads grow.
If money and supply were no object, HBM would be everywhere. But reality dictates that there are multiple tiers of memory and storage with tradeoffs between price and performance.
[View All](https://www.eetimes.com/category/sponsored-content/)
Low-power memory moves up the stack
LPDDR was purpose-built for battery-powered devices, such as smartphones, due to its low-power use, high enough bandwidth, and compact design. These same characteristics make it appealing for AI data centers.
The upcoming update to JEDEC’s LPDDR6 will squeeze in several features that address the memory’s increasing popularity in AI data centers, extending the memory standard beyond mobile platforms to support selected data center and accelerated computing workloads that need power-efficient, high-capacity memory.
Combined with the SOCAMM form factor, LPDDR delivers much higher memory capacity and energy efficiency near the CPU, closing the “memory wall” for token‑heavy models while lowering operating costs and thermal constraints.
SOCAMM modules place dense LPDDR dies in a compact, serviceable module that increases per‑socket capacity and bandwidth compared with traditional registered dual in-line memory module (RDIMMs), which enables larger KV caches and fewer off‑chip round trips for inference and retrieval use cases.
A SOCAMM consumes one-third of the power compared to standard DDR5 RDIMM; it’s also easier to stack together. The result is better performance-per-watt for inference and many production AI workloads.
Micron’s recently launched 256GB SOCAMM is designed for dense, serviceable server builds and supports much higher memory per CPU than traditional RDIMMs. It uses Micron’s 1-gamma DRAM process and monolithic 32Gb dies to boost capacity, improve power efficiency, and reduce time-to-first token for AI inference by off KV-cache from HBM.
“The opportunity for low-power DRAM in the data center is very high,” Praveen Vaidyanathan, VP and general manager of Cloud Memory Products at Micron, told EE Times in a briefing. “There are application spaces that are very well suited for the feature sets that this product provides.”
That capacity jump matters because AI workloads are shifting toward inference, where response time and memory footprint are becoming central design constraints with memory and compute working together to meet the need of these applications, Vaidyanathan said.
Emerging workloads such as mixture-of-experts models, multimodal AI, and agentic systems are increasing demand for memory close to compute, and long context windows are becoming more common. “Memory bandwidth is super important here,” Vaidyanathan said.
He added that AI systems are increasingly being optimized around the right mix of memory. “We’ve got to continuously be focused on efficiency, whether it’s performance, capacity, or power.”
HBM can’t do it all
Micron doesn’t see SOCAMM replacing HBM; rather, it sits in the middle of a broader memory hierarchy, with different memory tiers being assigned different roles as inference grows. “Hot KV cache remains in HBM, the warm KV cache remains in low-power DRAM, and the cold KV cache actually goes to fast storage,” Vaidyanathan said.
Jim Handy, principal analyst with Objective Analysis, said LPDDR and SOCAMM are gaining traction in data centers because of their capacity and fast access time—not because they were designed for AI.
HBM, meanwhile, is in high demand, which is made by stacking DRAM. “The HBM is spoken for because of AI, and of course that puts pressure on DRAM,” Handy said.
He said hyperscale operators are willing to pay for better efficiency because they do the math on electricity and total system cost—performance per watt is a decisive factor, especially when AI is consuming a large share of data center spending. “If we use this more expensive component, it’ll cut the power by a certain amount,” Handy said.
Rambus also sees a role for SOCAMM in the data center. The company recently announced its LPDDR-based SOCAMM2 chipset, the first in a planned family of LPDDR-based server module chipsets for future AI systems.
Like Micron, Rambus is working to support a memory hierarchy, Steve Woo, fellow and distinguished inventor at Rambus, told EE Times in an interview. “If you look at what’s going on in training, HBM is the big thing—people have these big foundation models,” Woo said. “When you get to the inference side, it’s a different story.”
Woo said AI memory is increasingly a story of fit, not just speed. You don’t need to spend money on HBM for inference when lower-cost options such as GDDR and DDR are sufficient. “There’s different levels and capabilities of AI, and different solutions will make more sense depending on what you’re willing to spend and the size and the models.”
This shift matters because the biggest AI workloads are getting too large to live in a single memory layer—memory tiering means the GPU holds the most frequently accessed data, while other data might get pushed down the hierarchy to SSDs.
Power is also a defining constraint, Woo said, noting that on high-performance GPUs more than half the power is spent moving the data back and forth to memory, which is why architectures, interconnects, and technologies such as the Compute Express Link (CXL) protocol are attracting so much attention.
For Woo, the economic logic is just as important as the technical one. Memory choices increasingly reflect who is paying for performance, how much they are willing to spend, and whether the workload is premium or price sensitive. AI dictates memory hierarchy
These realities are forcing data centers to rethink memory as a layered system rather than a single pool of high-speed memory.
Sandeep Krishnegowda, VP and general manager for memory at Infineon Technologies, said the answer to the memory bottleneck isn’t simply adding more of one device class—multiple tiers of memory are needed to ensure cost, power, and persistence across the AI stack.
HBM is obviously at the top of this hierarchy, he said, but SRAM also has a role to play serving the most latency-sensitive workloads, while DRAM, CXL pooled memory, and flash SSDs take on broader roles in context retention, configuration, and model storage. “There’s a lot of discussion around disaggregation of memory into compute racks,” Krishnegowda said.
He added that NOR flash may be one of the most overlooked enablers in the AI era, noting that it can be found throughout Nvidia GB200 racks in varied densities. “NOR is perfect to store the boot code and initialization of these server racks in an AI context.”
Krishnegowda also emphasized its role in security role by storing keys and certificates that verify hardware has not been tampered with. “It really becomes a hardware anchor in this space. I see multiple use cases of where NOR is very relevant today in an AI data center.”
Persistent memory finds its place
NOR flash may face competition from MRAM for code, firmware, inference metadata, and other persistent-but-not-archival data. Its non-volatility, combined with much faster writes, higher endurance, and lower power than NOR is suitable for AI systems that need frequent updates, fast recovery, and low-latency persistent state.
Sean Dougherty, VP of worldwide sales at MRAM maker Everspin Technologies, said the data center, the network, and then the edge will all need persistent memory support as AI spreads outward. He said discrete MRAM could play a strong role in AI at the edge, where it could alleviate memory bottlenecks in the data center by allowing more inference to be done locally.
He said MRAM’s immediate fit isn’t the biggest server rooms, given the existing mix of HBM, DRAM, and storage, but it does solve practical problems at the edge. “The ability to lose power and come back and be exactly in the same context you were when you went offline is extremely important.”
Dougherty said MRAM’s ability to execute in place is a key advantage for many embedded AI systems at the edge, highlighting extreme improvements in write time compared with NOR flash.
Everspin’s recently announced UNISYST MRAM unifies code storage and data memory in a high-density, non-volatile architecture for edge AI, industrial, and mission-critical designs. The new product family employs a unified code-and-data architecture that bridges traditional configuration memory and higher-density persistent storage to extend MRAM into traditional NOR flash applications.
Dougherty said that UNISYST reduces code writing in FPGA applications down to seconds compared to the 10 to 20 minutes of NOR flash. That’s a meaningful gap, he said, not only for boot and update latency but for simplifying code around wear-leveling. “That write time and write bandwidth starts to become really important as part of the overall system architecture.”
MRAM’s SRAM-like endurance and non-volatility makes it a compelling persistent-memory candidate as AI workloads continue to move toward the edge and into networked devices where power interruptions and latency matter most, Dougherty said.
Back in the data center, Handy said interconnects such as CXL can help with AI memory economics—CXL’s memory‑pooling can reduce stranded capacity and shift power allocation away from pin‑heavy memory interfaces.
He said the tradeoff is that software must be rewritten to place latency‑sensitive state on local DRAM and bandwidth‑hungry datasets on pooled memory. “You root the things that need really good latency to a local DRAM, and then you root everything else over to CXL.”
Read also:
[Canada’s AI Ecosystem Needs More Urgency](https://www.eetimes.com/canadas-ai-ecosystem-needs-more-urgency/)
[Defense Sends Clear Signal to Canadian Semiconductor Industry](https://www.eetimes.com/defense-sends-clear-signal-to-canadian-semiconductor-industry/)
[Canadian Researchers Reduce Quantum Atmospheric Turbulence](https://www.eetimes.com/canadian-researchers-reduce-quantum-atmospheric-turbulence/)