# [AI Ecosystem] The real bottleneck: Data, not compute > Source: > Published: 2026-09-02 23:59:52+00:00 As AI workloads evolve toward inference and agentic AI, the criteria for system performance are also changing. Computing power alone is no longer enough. Where data is stored, how quickly it moves, and how efficiently it is processed now have a direct impact on the performance and efficiency of AI systems. This article series explores the transformation of the AI ecosystem, where software, data center infrastructure, semiconductors, and memory technologies work together. It also examines why memory semiconductors have become a foundational technology for the AI era and outlines SK hynix’s vision for building the next-generation AI memory ecosystem. ① The paradigm shift in AI computing **② The real bottleneck: Data, not compute – Professor Hoi-Jun Yoo, KAIST** ③ Redesigning infrastructure: Why architecture determines performance ④ The semiconductor paradigm shifts toward memory ⑤ Completing AI: Physical intelligence and the role of memory As the center of gravity of AI shifts from pretraining to inference and from one-off responses to agentic AI, the location of system bottlenecks has also changed. The key issue increasingly lies in the flow of data rather than the volume of compute itself. As models maintain longer contexts and repeatedly reference intermediate results, the frequency of data exchanges between memory and compute devices increases. The question, therefore, is no longer limited to GPU compute capability. Is AI performance really constrained by how much a GPU can compute or by the memory architecture’s ability to retrieve the required data and deliver it to compute devices when needed? In the AI era, the core bottleneck lies not in compute itself, but in where data resides and how it moves through the system. ### The illusion created by the ‘GPU myth’: Why compute performance does not equal AI performance For the past decade, the formula for improving AI performance seemed clear: compute-centric scaling* that prioritizes more GPUs, larger models, and longer training runs. This formula held true during the era when large-scale pretraining was the primary path to improving AI performance. Deploying more GPUs made it possible to train larger models for longer periods, which in turn improved both model accuracy and versatility. As a result, GPUs became a symbol of performance in the AI era. Companies competed to highlight how many GPUs they had secured, the size of their clusters, and how many operations they could perform per second. These metrics clearly matter. Without the parallel compute capabilities of GPUs, today’s large-scale AI models would be difficult to achieve. However, the real-world performance of an AI system is not determined solely by a GPU’s theoretical compute capability. No matter how many compute devices a system has, it cannot deliver its full performance if the required data does not arrive in time. In other words, what matters is not only how quickly a GPU can compute — the required data must also be located close to the compute device and accessed without delay for the GPU’s performance to be fully realized. This distinction has become more pronounced with the proliferation of inference-centric AI and agentic AI. During large-scale training, compute devices could be kept relatively well utilized by processing large amounts of data together in batches*. In contrast, real-world inference environments require low latency, long contexts, and repeated calls. Models repeatedly reference required information, store intermediate results, and continuously retrieve data to generate the next token*. * Batch: A unit in which an AI model processes multiple pieces of data together. Large batches can help utilize GPUs efficiently during large-scale training but are more difficult to use for inference in real-world services because of low-latency requirements and the need to process individual requests As a result, the focus of the bottleneck is shifting from GPU compute capability to the location and movement path of data. The key challenge is how to overcome the limits of memory bandwidth and access latency — the memory wall*. ### The memory wall: A 30-year-old warning becomes reality in the AI era In 1994, U.S. computer scientists William A. Wulf and Sally A. McKee highlighted the growing gap between processor performance and memory access speed in a short paper*. They warned that while processor performance was improving by around 60% each year, DRAM access speed was improving by only about 7%. If that gap continued to widen, system performance could ultimately become constrained by memory speed. This bottleneck became known as the memory wall. Thirty years later, that warning is becoming a reality in AI service environments where large models with tens or hundreds of billions of parameters operate in real time. While the compute performance of the latest AI accelerators processing these models has reached the petaflops* range, the speed at which model weights and activations can be retrieved from memory has not kept pace with advances in compute. As a result, many of the compute units integrated into large chips spend more time waiting for data, reducing their actual utilization. There are two primary approaches to narrowing this gap: increasing memory bandwidth itself or placing memory closer to the processor to reduce the distance data must travel. ### The inference era: KV cache is taking over GPU memory As large language models* and reasoning-intensive workloads become widespread, memory bottlenecks have become more complex. Unlike earlier, one-off inference tasks, today’s models generate thousands — sometimes tens of thousands — of tokens sequentially in response to a single query. Step-by-step reasoning approaches such as chain-of-thought* and agentic AI can push token counts even higher. * Chain-of-thought(CoT): A prompting and generation technique in which a model works through intermediate reasoning steps before producing an answer, helping improve reasoning accuracy One of the biggest challenges in this process is the KV cache*. Each time a Transformer*-based model generates a token, it retains key information from all preceding tokens in memory as key and value vectors. As the sequence grows, so does the cache. In some long-context inference environments, the memory required of the KV cache can even exceed the roughly 140 GB required to hold the weight of a 70-billion-parameter model. * Transformer: A neural network architecture introduced by Google in 2017. It uses self-attention to understand context and forms the foundation of nearly all modern LLMs. Put another way, when a user sends a query to an LLM service such as ChatGPT, more than half of the GPU’s work can involve moving data rather than computing it. This is a structural challenge that cannot be solved simply by adding more GPUs. ### Software’s clever fixes — and their emerging limits To ease these bottlenecks, the industry has adopted a range of software-based solutions. One of the most prominent is FlashAttention*. This technique restructures attention*, a core operation in LLMs, by dividing it into smaller blocks and processing as much as possible within the GPU’s internal SRAM*. This reduces repeated reads and writes of intermediate data between HBM and SRAM, improving inference efficiency on the same hardware. Techniques such as PagedAttention* and speculative decoding*, which optimize memory usage patterns and token generation, have also quickly gained traction. * Attention: The process by which an AI model refers to preceding context when generating the next word * Static random-access memory(SRAM): High-speed memory used for caches inside CPUs and GPUs. It is faster than DRAM but requires more circuit area, making it unsuitable for high-capacity storage. In AI compute, SRAM is used to keep frequently accessed intermediate data close to the processor * PagedAttention: A technique that manages GPU memory in pages, similar to virtual memory, during attention operations. By reducing unnecessary memory waste, it enables more user requests to be processed simultaneously and improves inference efficiency * Speculative decoding: An inference acceleration technique in which a smaller auxiliary model predicts multiple tokens in advance and the main model verifies them together. It is used to improve response speed while maintaining generation quality. Another major trend is quantization*. This method represents and stores model weights, activation values, or the KV cache using fewer bits, such as 8 or 4 bits instead of 16. With less data to represent, both memory usage and data movement decrease. Recently, various compression techniques have emerged to improve data representation efficiency, such as data formats that group similar values together to enhance representation, and vector quantization*, which replaces frequently occurring patterns with shorter codes. One recent development attracting attention in the field is TurboQuant*, a quantization technique that represents the KV cache with fewer bits to reduce its memory footprint. Conventional compression techniques have had to recalibrate their compression rules whenever the distribution of input data changes. While this technique offers high accuracy, it also increases processing time. TurboQuant takes a different approach. It first applies a mathematical operation known as a rotation to transform different inputs into similar distributions, then applies a set of predefined rules. The idea is similar to standardizing customers’ body shapes before fitting them with the same size of clothing, rather than tailoring a new outfit for every customer. According to the research, TurboQuant reduces a KV cache stored at 16 bits to around 3-4 bits, lowering memory usage to roughly one-fifth without significantly compromising response quality. This TurboQuant technique pushes compression efficiency — representing the same data with fewer bits — close to its theoretical limit. In other words, there may be limited room to compress the data much further without information loss. There is a key implication here. As software compression approaches its theoretical limits, the potential for further improvement within the same memory architecture also diminishes. Software optimization is clearly effective, but fundamental limitations remain if compute devices must continue fetching data from memory located farther away. ### The center of gravity is shifting toward memory So what comes next? The starting point is to bring memory and compute closer together, creating architectures that process the data they need over shorter paths. This is not simply a matter of finding better algorithms. It is a problem that requires redesigning the architecture of systems and semiconductors themselves. Several solutions have already emerged. One prominent example is High Bandwidth Memory*. HBM stacks multiple DRAM dies vertically and places them close to the GPU, shortening the distance data must travel to reach the compute device while increasing the amount of data that can be transferred at once. HBM is drawing attention not simply because it increases memory speed, but also because it reduces the time compute devices spend waiting for data. This same critical perspective is driving other memory and system technologies. High Bandwidth Flash* seeks to secure both greater capacity and higher bandwidth, while Compute Express Link* reflects a shift toward turning memory from a dedicated resource for a specific GPU into a shared resource at the system level. Processing-in-memor* goes a step further, processing some compute closer to the memory where data resides. Although these three technologies differ in their approaches, they all stem from the same question: Should data continue traveling long distances for processing, or should more work be processed closer to where the data is located? * Compute Express Link(CXL): A next-generation interconnect standard that connects CPUs, GPUs, memory, and accelerators through a coherent cache protocol, enabling memory to be disaggregated and shared at the system level * Processing-in-memory(PIM): A next-generation memory technology that adds processing capabilities to memory, helping reduce data bottlenecks between memory and processors and significantly improve performance Ultimately, these approaches point in the same direction. Data that can be kept close to the compute device should stay close, while data that must travel requires wider, more efficient paths. The AI bottleneck is no longer a problem that can be addressed at the level of a single chip. It is a system-level challenge involving the complex connections between model architecture, GPUs, memory, interconnects, packaging, and data center architecture. This does not mean that conventional von Neumann architecture*, which separates compute devices from memory, is about to disappear. It does mean, however, that an approach taken for granted for decades — keeping processors and memory separate and connecting them through a bus — must now be reexamined. AI system performance is determined not only by the compute device itself, but also by where memory is located and how data is read. ### Where is the invisible bottleneck? Let us return to the initial question: What is the real bottleneck in AI? It lies in a place that may not be immediately visible — in an architecture where data constantly moves back and forth between memory and compute devices. The software-based compression techniques discussed earlier can alleviate some of this bottleneck, but a fundamental question remains. Why must information travel such a long path every time? Can more processing be done where the data resides? Can the gap between memory and compute be narrowed? How effectively this bottleneck is addressed will determine the competitiveness of next-generation AI infrastructure. This is not a challenge limited to any single country or company. It is a challenge for the global AI ecosystem, requiring AI accelerators, memory, advanced packaging, interconnects, and software to work together. Within this context, memory technology capabilities have become more critical than ever. This is ultimately where the next phase of competition in AI infrastructure will be decided. Simply securing more compute devices will not be enough. The advantage will go to those that can co-design memory layouts and compute structures to process data faster and more efficiently. Understanding this shift and translating it into new architectures is one of the most important tasks facing semiconductor researchers today. ### The next question: What architecture can solve the AI infrastructure bottleneck? The bottleneck examined in Part 2 is not simply a matter of memory speed. As inference and agentic AI become more widespread, models must maintain long contexts, repeatedly reference intermediate results, and continuously exchange the necessary data between memory and compute devices. As a result, AI performance increasingly depends less on the compute capability of an individual GPU and more on where data is stored, how it moves through the system, and how efficiently it is processed. This challenge is difficult to solve with an individual chip or memory technology alone. GPUs, memory, storage, networks, and data centers must work together as a single system. In the next installment, we examine this challenge from an AI infrastructure perspective. What ultimately determines AI performance: faster components or the overall architecture through which compute and data flow? **Disclaimer:** The opinions expressed in this article are solely those of the author and do not necessarily reflect the official position of SK hynix.