For years, the AI infrastructure story has been dominated by training—bigger models, larger GPU clusters, and data centers built to absorb the computational load. But as those models move into large-scale use, inference is turning connectivity into one of the central bottlenecks in AI system design. “Inference is the moment when users use the model,” Peter Ossieur, portfolio director at imec and professor at Ghent University, told EE Times. “So, it needs to be reactive.”
That makes inference a different problem from training. Training large models is enormously compute-intensive, but it is less directly tied to the end user’s experience of waiting for an answer. Inference must generate tokens quickly and repeatedly, often for many users at once, involving long context windows, retrieval-augmented generation, reasoning steps, multimodal inputs, and multi-agent architectures.
Inference is not just a compute problem. It’s a memory, network, bandwidth, latency, and energy problem.
View All Scale-up becomes the new bottleneck
The key distinction is between scale-out and scale-up networking.
Scale-out networks connect racks and systems across a data center. That domain is already optical and has driven much of the demand for pluggable optical transceivers. Scale-up is more demanding. It connects GPUs or other accelerators tightly enough that, from the application’s point of view, they behave like one very large processor.
Today, that scale-up fabric is still often copper-based and largely constrained to the rack. But that model becomes harder to sustain as AI systems move toward hundreds, thousands, or even tens of thousands of accelerators acting together.
The traffic stems from frontier AI models exceeding the capacity of a single GPU and its nearby high-bandwidth memory. Training and inference tasks must therefore be split across multiple accelerators, resulting in intensive GPU-to-GPU communication. That problem could worsen as AI systems become more agentic, with specialized agents exchanging data among themselves and with larger foundational models.
For imec, the result is a system-level interconnect challenge. If thousands of GPUs need to communicate with low latency, the network architecture must minimize hops. That points toward high-radix switches and potentially new optical switching approaches, including optical circuit switching. “You need to co-design the software, the system architecture, and then the physical layer as a whole,” Ossieur said. “You need to make sure that all of these work together and are optimized for that AI application that you’re trying to address.”
From co-packaged optics to 3D optical I/O
The first major step in moving optics closer to compute is co-packaged optics. Instead of placing optical modules away from the processor or switch, co-packaged optics brings the optical engines into the package, shortening the electrical path and improving bandwidth and energy efficiency.
For scale-up, it may become essential. But Imene Jadli, portfolio manager for optical interconnect at imec, argued that co-packaged optics will not be enough for future AI systems. “It’s a logical next step, but not the end of the roadmap,” she told EE Times. The reason is power. In one projection discussed by imec, a future processor could require roughly 250 Tb/s of bandwidth in and out. If that bandwidth were supplied using anticipated co-packaged optics approaches, the optics alone could consume about 1.25 kW, on top of processors that may already consume several kilowatts.
For system designers, that becomes a packaging and thermal problem. If the optics added to the package consume too much power, the system may exceed what can realistically be cooled. “We need something that’s far better than co-packaged optics,” Ossieur said.
Imec’s proposed direction is 2.5D optical I/O. In that approach, optics moves closer to the processor by being integrated at the interposer or substrate level. The goal is to shorten the electrical path while reducing the power required to move massive amounts of data.
A key part of the concept is what Ossieur called a “wide and slow” approach. Rather than using a small number of extremely high-speed lanes with sophisticated signal processing, imec is looking at many more lanes operating at moderate speeds. The aggregate bandwidth can remain high, but the energy per bit can fall.
In the same future system projection, Ossieur said 2.5D optical I/O could reduce optical power from about 1.25 kW to below 200 W.
That would make the optical interconnect more manageable from a thermal and system-design standpoint. But it also requires advances across optical devices, packaging, assembly, materials, and system architecture.
For co-packaged optics, Jadli said, the industry needs compact devices with high efficiency and high bandwidth, including electro-absorption modulators and high-speed photodetectors. As the roadmap moves toward 2.5D and 3D optical I/O, device efficiency and integration become even more central. Silicon remains attractive because it’s compatible with CMOS manufacturing. But silicon photonics alone may not provide all the properties needed for future optical I/O. Imec is therefore exploring materials such as barium titanate and III-V compounds, along with new integration schemes that can bring those materials into a CMOS-compatible flow.
The longer-term endpoint is 3D optical I/O, where optics becomes a native part of the 3D compute stack. That could eventually support compute-to-compute or memory-to-compute communication inside advanced packages, including links involving high-bandwidth memory and stacked processors.
But the closer optics moves to the processor, the more difficult the integration challenges become. Materials compatibility, manufacturing yield, optical coupling, testing, and ecosystem readiness all matter. For 3D optical I/O, Jadli identified thermal management as one of the hardest unresolved problems.
That engineering tension is what makes the roadmap significant. The industry may need optics closer to compute to keep AI scaling, but moving optics closer to compute means solving problems across photonics, packaging, materials science, and system architecture.
As inference becomes the dominant workload for large AI systems, the question is no longer whether optics will play a role in AI infrastructure. It’s how close to the processor optics will have to go.
Read also:
[Lightmatter Unveils 3D Co-Packaged Optics for 256 Tbps in One Package](https://www.eetimes.com/lightmatter-unveils-3d-co-packaged-optics-for-256-tbps-in-one-package/)
[Ayar Labs and Alchip Unveil Optical I/O Reference Design](https://www.eetimes.com/ayar-labs-and-alchip-unveil-optical-i-o-reference-design/)
[Lumai Productizes Lens-Based Optical Computer](https://www.eetimes.com/lumai-productizes-lens-based-optical-computer/)