# [AI Infrastructure Insight] Why faster GPUs alone can’t deliver AI performance

> Source: <https://news.skhynix.com/en/ai-infrastructure-insight-ep2/>
> Published: 2026-08-30 23:59:33+00:00

AI is no longer defined by a single model or chip. For AI to operate effectively in real-world services and industrial applications, it takes faster compute, greater memory bandwidth, higher-performance networking, more efficient storage, and stable power and cooling working together. This is why competition in AI is shifting beyond individual technologies to the design and operation of the entire infrastructure.

In the** AI Infrastructure Insight** series, we explore how the architecture behind AI infrastructure is evolving — from compute, memory, storage, networking, and power and cooling to the way they come together as a unified system — and what this means for the industry.

**[Series overview]**

① What is changing inside AI data centers?

**② Why faster GPUs alone can’t deliver AI performance**

③ Why power and cooling have become the next challenge for AI infrastructure

④ How AI infrastructure will be designed for the future

Does using a faster GPU* result in faster AI operations? GPUs and AI accelerators* are core technologies that have enabled the training of large-scale models and the processing of complex inference requests. In real-world service environments, however, AI performance is not determined by compute speed alone.

* AI accelerator: A semiconductor or computing device designed to rapidly perform the large-scale computations required for AI training and inference. Examples include GPUs, NPUs, and TPUs

Even when a processor is ready to perform calculations, an expensive accelerator can be left waiting if data is not supplied in time, reducing overall system efficiency.

An AI system can therefore deliver its full performance only when accelerator compute capability, data delivery speed, and data movement paths work together effectively.

The research supports this point.

The paper “AI and Memory Wall,”* by researchers from UC Berkeley, ICSI, and LBNL, found that over the past 20 years, peak FLOPS* in server hardware increased by approximately threefold every two years, while DRAM bandwidth* increased by only 1.6 times and interconnect* bandwidth by 1.4 times.

While compute performance has advanced rapidly, the ability to supply and move data has progressed at a comparatively slower pace. As this gap widens, bottlenecks* in memory and data movement paths — not just compute performance — may become more pronounced in AI systems.

* Peak floating point operations per second(FLOPS): The theoretical maximum floating-point compute performance of a computing system. FLOPS refers to the number of floating-point operations that can be performed per second

* Bandwidth: A measure of the amount of data that can be transferred over a given period of time. It is commonly used to describe data transfer performance in memory and networks.

* Interconnect: An architecture or technology that connects components such as servers, chips, memory, and accelerators, enabling data to move between them

* Bottleneck: A point that limits overall system performance. In AI systems, bottlenecks can occur not only in processors but also in data delivery, memory, storage, and networks

### Data never stays in one place

In an AI system, data does not remain stagnant in one place. Training data, model parameters*, user requests, contextual information, search results, and compute results move between different locations depending on the stage of processing. Some data is retrieved from high-capacity storage, while other data passes through system memory before reaching memory closer to the accelerator. Once processing is complete, the results are stored again or transmitted over the network to other servers and services.

☑︎ Where does data travel before AI answers?When a user enters a question into a chatbot or AI search engine, the data center processes more than just the user’s request. Previous conversational context, along with search results, relevant documents, and other information the model may need, is gathered as required. This data moves through storage and memory before reaching layers closer to the accelerator. Once computation is complete, the result is stored again or returned to the user over the network.

As the use of AI search and chatbots grows, these inference requests are occurring more frequently and at greater scale. Ultimately, the response speed experienced by users depends not only on model performance but also on how quickly the necessary data can be gathered and moved.

As of July 2025, 18 billion messages were exchanged on ChatGPT each week, according to OpenAI, while Gemini surpassed 1 billion monthly users after August 2026.

Within this flow, each layer plays a different role. Storage retains large volumes of information, while system memory provides a workspace for the entire server. Memory located close to the accelerator rapidly supplies the information needed by the processor, while networks connect multiple servers and accelerators into a single system.

AI performance, therefore, is not guaranteed simply by having the necessary data available. Even when using the same GPU, systems can deliver different real-world performance depending on how efficiently data is supplied and moved.

### Bottlenecks beyond compute

Bottlenecks in AI systems do not occur in a single location. No matter how fast a processor is, insufficient memory bandwidth can prevent the system from supplying enough data when it is needed. In this case, an accelerator may be ready to compute but still spend time waiting for data, reducing actual utilization rates.

Storage can also become a bottleneck* point. When an AI service needs to retrieve large volumes of documents, images, logs, user histories, or search results, response speeds slow down if the required data cannot be read quickly enough.

As the number of requests for inference services grows, such delays can directly degrade service quality.

Similar challenges arise during large-scale AI training. Training an AI model requires checkpointing*, which periodically saves the model’s intermediate training state. If storage fails to save or retrieve these large volumes of data quickly enough, latency can occur across the entire training cluster. This is why MLCommons* separately measures storage performance for AI and machine learning training workloads through its MLPerf Storage* benchmark. Data management is a core element of AI training and inference, making storage performance an increasingly important part of AI infrastructure.

* MLCommons: A global AI engineering consortium that develops industry-standard benchmarks for measuring the performance and efficiency of AI systems

* MLPerf Storage: A suite of MLCommons benchmarks that evaluates the performance of storage systems supporting AI and machine learning workloads. It measures how data reads and delivery, as well as checkpoint saving and recovery during AI training, affect system efficiency

Networks are another potential source of bottlenecks. In large-scale AI training or high-performance inference, multiple servers and accelerators exchange data simultaneously to divide and process a single task. If data transmission between servers are delayed or network bandwidth is insufficient, slowdowns in one part of the system can reduce overall processing speed. It is not enough for an individual device to be fast. Multiple devices need to exchange data efficiently and operate as a unified system.

Power and cooling can also constrain performance. High-performance equipment requires a stable power supply and effective thermal management to sustain peak performance. Part 3 of this series will explore these issues in greater detail, but the key point is clear: Bottlenecks and performance limitations can arise throughout the system — from data movement paths involving memory, storage, and networks to power and cooling infrastructure.

### Data architecture improves accelerator utilization

The key to AI infrastructure lies not only in securing more accelerators but also in designing systems that keep those accelerators continuously engaged in computation. When data is supplied too slowly, accelerator utilization* falls, reducing performance relative to the investment made. This is why even systems equipped with high-performance hardware may fail to perform at their full potential.

To prevent this, data paths need to be designed so that data can be supplied where and when it is needed. Frequently used data should be placed close to processors, while large volumes of data should be retrieved efficiently through storage and networks. This is due to the fact that it is impossible to place all information in the fastest location. Ultimately, the location and movement paths of data must be meticulously designed with speed, capacity, cost, and power efficiency in mind.

Software coordinates this architecture for real-world operations. Accelerator utilization and overall performance vary depending on which tasks are processed first, when data is retrieved, and how they are both distributed across multiple devices. Even with fast hardware, inefficient scheduling and placement can create new bottlenecks.

Designing to minimize data movement can also improve power efficiency. Intel explains that while the process of moving data within a system consumes energy, it does not directly contribute to computation itself. This is why data should travel shorter distances and be placed closer to where it is needed.

Within this architecture, memory serves as a critical layer that supplies data where and when it is needed, helping accelerators remain continuously engaged in computation. How efficiently compute and memory are connected can therefore determine real-world system performance.

### AI infrastructure competitiveness depends on data flow

This approach of designing systems around data paths is bringing about changes in the competitive landscape of the AI infrastructure industry. While fast GPUs, high memory bandwidth, high-capacity storage, and high-speed networks are all important individually, they are insufficient if they operate in isolation. AI systems can deliver real-world performance only when data moves smoothly from storage to the processor, and results are delivered where they are needed.

Consequently, collaboration methods across the entire industry ecosystem are also shifting. There are limitations to maximizing overall system performance if semiconductor, server, networking, cloud, and storage companies each optimize only their own components independently. As AI workloads grow larger and more complex, collaboration based on the entire data flow becomes crucial. This is due to the fact that required infrastructure architecture varies depending on which models customers use, what data they process, and what level of response speed is required.

The role of memory companies is expanding as well. Beyond merely supplying products, they increasingly need the capabilities to jointly design data flows within customer systems and determine what memory architectures are required from a system-level perspective encompassing processors, storage, and networks.

Compute remains essential. Faster GPUs and AI accelerators will continue to be core components of AI infrastructure. However, translating their capabilities into real-world performance requires optimizing the flow of data across the system as well.

Looking ahead, understanding AI infrastructure requires examining not only the speed of processors but also the ways in which data moves through the system. Memory, storage, networks, and power and cooling should not be considered as independent components but as organically interconnected parts of a single system that keep data flowing without interruption.

The need for faster GPUs remains as essential as ever. But the question has now evolved one step further: How quickly and efficiently does the data reach the GPU for processing?

The next bottleneck determining AI performance begins with that question.

**<References****>**

- Amir Gholami et al., “
[AI and Memory Wall](https://arxiv.org/abs/2403.14123),” IEEE Micro, Vol. 44, No. 3, 2024. - OpenAI, “
[How People Use ChatGPT](https://cdn.openai.com/pdf/a253471f-8260-40c6-a2cc-aa93fe9f142e/economic-research-chatgpt-usage-paper.pdf),” 2025. - Google, “
[Google’s Gemini app hits 1 billion monthly active users](https://blog.google/innovation-and-ai/products/gemini-app/one-billion-monthly-users/),”

Google Blog, Aug. 11, 2026.