{"slug": "ais-hidden-energy-tax-more-than-80-of-scale-up-traffic-is-just-memory-moving", "title": "AI’s hidden energy tax: More than 80% of scale-up traffic is just memory moving around", "summary": "More than 80% of scale-up network traffic in AI data centers is memory transactions, and moving HBM into shared photonic pools can cut data-movement energy to a quarter of copper-based approaches, Marvell Technology Photonic Fabric Business Unit senior vice president and general manager Preet Virk said at SC26. Penguin Solutions vice president of advanced product development Andy Mills described Penguin's MemoryAI Photonic Appliance, which holds 16 terabytes of HBM and DDR5 memory per unit and links to GPUs tens of meters away via Marvell's Photonic Fabric so servers across multiple racks can share it. The executives pitched shared memory pools as the answer to low GPU utilization and KV-cache recomputation as inference and agent workloads spread across more racks.", "body_md": "### AI’s hidden energy tax: More than 80% of scale-up traffic is just memory moving around\n\nMemory bottlenecks are emerging as one of the costliest problems in AI infrastructure as inference overtakes training. GPUs sit idle waiting on data, while high-bandwidth memory, or HBM, stays bolted to processors that cannot share it.\n\nPhotonics is shaping up as one way out. Marvell Technology Inc. has been [pairing optics with photonic memory](https://siliconangle.com/2026/10/26/ai-architectures-marvell-data-center-thecube-aifactoriesdatacenters/) to stretch memory beyond the edge of a chip. The payoff is better efficiency in data centers enterprises have already paid for, according to [Preet Virk](https://www.linkedin.com/in/preetvirk/) (pictured, left), senior vice president and general manager of the Photonic Fabric Business Unit at Marvell.\n\n“In the data center today, we have a lot of room for improvement of efficiency,” Virk said. “And when I say efficiency, I’m alluding to the wasted flops because we have GPU utilization that’s very low.”\n\nVirk and [Andy Mills](https://www.linkedin.com/in/amillzie/) (right), vice president of advanced product development at Penguin Solutions Inc., spoke with theCUBE Research’s [John Furrier](https://www.linkedin.com/in/furrier/) and [Amit Eyal Govrin](https://www.linkedin.com/in/amitgovrin/) at [SC26](https://www.thecube.net/events/acm-ieee/sc26), during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed how photonics and shared memory pools can ease memory bottlenecks in AI infrastructure. *(* Disclosure below.)*\n\n### Shared pools take aim at memory bottlenecks\n\nPenguin’s [MemoryAI Photonic Appliance](https://ir.penguinsolutions.com/news/news-details/2026/Penguin-Solutions-Unveils-Industrys-First-Photonics-Based-HBM-Class-Memory-Appliance-to-Break-the-GPU-Memory-Wall-for-Agentic-AI/default.aspx) moves HBM into a separate box tens of meters from the GPU, linked by Marvell’s Photonic Fabric. Each appliance holds 16 terabytes of HBM and DDR5 memory that servers across multiple racks can share, Mills explained.\n\n“The challenge there is making sure your LLM doesn’t forget what you started talking to it about 10 minutes ago,” Mills said. “To do that, you need a large amount of memory, large token lengths.”\n\nOptics also changes the energy math. Copper still works inside a rack, but crossing a pod of thousands of GPUs requires optics, according to Virk. More than 80 percent of scale-up network traffic is memory transactions, and a shared pool means much of that data never moves at all, he noted.\n\n“Within data movement, the memory transactions are a place to focus on,” Virk said. “So with this solution versus a copper-based data movement, we could immediately get to a quarter of the energy, picojoules per bit, of data movement.”\n\nAgents raise the stakes, generating far more token requests than human users ever did. As inference spreads across more racks and sites, Mills sees shared pools as the practical answer to memory bottlenecks.\n\n“If anything, distributed encourages shared because you don’t want to recompute,” Mills said. “The whole thing about KV cache was a huge success because it stops recomputation of the same data. And when you can share that data on a broader base and wider across more racks, your efficiencies go up as well.”\n\nHere’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of [SC26](https://www.thecube.net/events/acm-ieee/sc26):\n\n*(* Disclosure: Penguin sponsored this segment of theCUBE. Neither Penguin nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)*\n\n##### Photo: SiliconANGLE\n\n# A message from John Furrier, co-founder of SiliconANGLE:\n\nSupport our mission to keep content open and free by engaging with theCUBE community. **Join theCUBE’s Alumni Trust Network**, where technology leaders connect, share intelligence and create opportunities.\n\n- **15M+ viewers of theCUBE videos** , powering conversations across AI, cloud, cybersecurity and more\n- **11.4k+ theCUBE alumni** — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network\n\n### Are you an AWS customer?  Support SiliconANGLE financially by buying your AWS services from our Marketplace portal page and links: [https://siliconangle.com/aws-marketplace/](https://siliconangle.com/aws-marketplace/)\n\n##### **About SiliconANGLE Media**\n\n[SiliconANGLE](https://cts.businesswire.com/ct/CT?id=smartlink&url=https%3A%2F%2Fsiliconangle.com%2F&esheet=54119777&newsitemid=20240910506833&lan=en-US&anchor=SiliconANGLE&index=9&md5=646b1b564e2259100a2b8638aab0a552),\n\n[theCUBE Network](https://cts.businesswire.com/ct/CT?id=smartlink&url=https%3A%2F%2Fwww.thecube.net%2F&esheet=54119777&newsitemid=20240910506833&lan=en-US&anchor=theCUBE+Network&index=10&md5=7de2a85f95ab4a4a495cede20b8cb1da),\n\n[theCUBE Research](https://cts.businesswire.com/ct/CT?id=smartlink&url=https%3A%2F%2Fthecuberesearch.com%2F&esheet=54119777&newsitemid=20240910506833&lan=en-US&anchor=theCUBE+Research&index=11&md5=7bb33676722925eb57d588ec343e4f6f),\n\n[CUBE365](https://cts.businesswire.com/ct/CT?id=smartlink&url=https%3A%2F%2Fwww.cube365.net%2F&esheet=54119777&newsitemid=20240910506833&lan=en-US&anchor=CUBE365&index=12&md5=d310fb35919714e66ad8d42c9c0c1bc6),\n\n[theCUBE AI](https://cts.businesswire.com/ct/CT?id=smartlink&url=https%3A%2F%2Fwww.thecubeai.com%2F&esheet=54119777&newsitemid=20240910506833&lan=en-US&anchor=theCUBE+AI&index=13&md5=b8b98472f8071b23ebb10ab9a8dd0683)and theCUBE SuperStudios — with flagship locations in Silicon Valley and the New York Stock Exchange — SiliconANGLE Media operates at the intersection of media, technology and AI.\n\nFounded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.", "url": "https://wpnews.pro/news/ais-hidden-energy-tax-more-than-80-of-scale-up-traffic-is-just-memory-moving", "canonical_source": "https://siliconangle.com/2026/11/18/memory-bottlenecks-ai-how-photonics-shared-pools-help-sc26/", "published_at": "2026-11-18 14:57:55+00:00", "updated_at": "2026-11-18 15:13:47.266555+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-chips", "ai-agents", "mlops"], "entities": ["Marvell Technology Inc.", "Penguin Solutions Inc.", "Preet Virk", "Andy Mills", "MemoryAI Photonic Appliance", "Photonic Fabric", "SC26", "theCUBE"], "also_reported_by": [], "takeaway": "Pooling HBM and DDR5 in a shared photonic appliance 16 terabytes per unit and tens of meters from the GPUs avoids much of the memory traffic that makes up over 80% of scale-up network transactions, cutting data-movement energy to a quarter of copper's picojoules per bit.", "recommendation": {"advice": "When scaling inference across racks, share KV cache and HBM through a pooled photonic memory appliance instead of recomputing or moving memory over copper; the source cites a quarter of the energy per bit.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/ais-hidden-energy-tax-more-than-80-of-scale-up-traffic-is-just-memory-moving", "markdown": "https://wpnews.pro/news/ais-hidden-energy-tax-more-than-80-of-scale-up-traffic-is-just-memory-moving.md", "text": "https://wpnews.pro/news/ais-hidden-energy-tax-more-than-80-of-scale-up-traffic-is-just-memory-moving.txt", "jsonld": "https://wpnews.pro/news/ais-hidden-energy-tax-more-than-80-of-scale-up-traffic-is-just-memory-moving.jsonld"}}