TL;DR — Key Takeaways
More GPUs do not guarantee more performance. At scale, synchronization, latency and network efficiency determine how much of that compute is actually usable.MoE and heterogeneous AI shift pressure onto the network. Dynamic routing and hardware handoffs create less predictable traffic and new bottlenecks.AI infrastructure must be designed as one system. Compute, network topology, communication libraries and model architecture need to be optimized together.
AI development has become one of the largest capital stories in the market. Anthropic’s multi-billion-dollar valuation, OpenAI’s potential IPO, and SpaceX’s valuation tie-in with xAI all show that the market currently rewards the ability to build and operate massive compute clusters.
At today’s scale, large models’ performance is no longer just about how many GPUs a company can deploy. Actual output depends on how effectively those GPUs are connected, synchronized, and kept utilized.
The scaling equation has changed due to three main shifts:
Large-Scale clusters: scaling workloads across thousands of individual nodes.** Mixture-of-Experts (MoE) Architectures**: Activating only specific parts of a model per token, which reduces compute needs.** Heterogeneous AI:**Using different types of hardware to optimize specific parts of the inference pipeline.
While these trends improve efficiency at the compute layer, they also shift the bottleneck to the network. Ultimately, no matter how many and what type of GPUs are added to a cluster, overall performance remains limited by how well those GPUs are connected and synchronized.
The Limits of the Compute-First Mindset
In traditional dense models, every token activates every parameter in the network. A 400-billion-parameter model means every token triggers 400 billion calculations. The compute cost is enormous.
But these models also have a hidden advantage: Predictability. Engineers know which GPUs need to exchange information, when that exchange needs to happen, and how much data needs to move. This allows the network to be optimized around structured collective operations such as All-Reduce, All-Gather, and Reduce-Scatter.
These models communicate like an orchestra. Every GPU knows the next move before the first token starts.
The real challenge begins when scaling out across more and more nodes. More nodes mean more communication, more synchronization, and more pressure on the network. At massive scale, the network becomes one of the main factors that determines whether the cluster delivers on its promised performance.
This is where the “just add more GPUs” approach breaks. If the network cannot feed, synchronize, and coordinate the GPUs efficiently, expensive compute sits idle, waiting for data.
Mixture-of-Experts changed the equation
One question challenged the basic assumption behind dense models. Does every token really need to activate every parameter?
That insight gave rise to Mixture-of-Experts. Instead of one general neural network handling every token at full weight, MoE introduces specialized expert networks and a lightweight router. For each token, the router decides which experts should be activated.
Think of it as assigning each task to the right specialist. A coding-related prompt may activate experts tuned for software logic, while a graphics-related prompt may activate experts tuned for visual concepts. The model still has broad capacity, but only the most relevant parts are used for each token.
Because only the most relevant experts are activated for each token, MoE models can store far more knowledge than a dense model while using a fraction of the compute per request.
This is why MoE is so attractive for large LLM companies like Anthropic and OpenAI. It gives their AI teams a way to scale model capacity without scaling compute at the same rate.
But this efficiency comes with a tradeoff. MoE moves the bottleneck from compute to the network.
Where dense models communicate like an orchestra, MoE models communicate more like a group of improvisers. Each token may follow a different path, and routing decisions happen at runtime based on the input itself.
This creates dynamic All-to-All communication patterns, leading to unpredictable traffic, hot spots, and transient congestion on specific paths. Unlike dense models, the network cannot be pre-tuned for static collective operations because routing happens dynamically for every token.
As a result, MoE solves part of the massive compute problem, but it makes the network even more important.
Specialization Beyond the Model
MoE introduced the idea that not every part of a model needs to be active for every task. The same logic now extends to the hardware itself. Different phases of an AI workload have fundamentally different resource requirements, and running all of them on the same type of processor is the hardware equivalent of a dense model. It is capable and uniform, but increasingly inefficient as workloads scale.
Inference makes this clearest. The prefill stage is compute-heavy and demands raw processing throughput to handle the input prompt. The decode stage is memory-bandwidth-heavy and generates the next output token. These two phases need different hardware capabilities, and when both run on the same processor type, one phase usually gets the resources it needs while the other does not.
But separating the hardware creates a new network dependency. When prefill completes, the KV cache it built needs to move to the decode processor before generation can begin. The faster that handoff happens, the more the hardware specialization pays off. The slower it is, the more idle time erases the gain.
Heterogeneous clusters can improve compute efficiency and reduce cost, but they also generate traffic that is harder to predict. Once again, the network becomes the factor that determines whether workloads can deliver the promised performance gains.
Networking has Become the Performance Ceiling
Each step in AI infrastructure evolution has added new pressure on the network. Dense models at scale needed a fabric that could synchronize thousands of GPUs without becoming the bottleneck. MoE added unpredictability, producing dynamic and uneven traffic that dense models never generated. Heterogeneous AI adds new data handoff demands and even more chaotic traffic patterns.
The trend is consistent. As AI systems get more capable and more efficient, the demands on the network grow rather than shrink. Idle compute is the most expensive outcome. More hardware diversity creates more coordination. More efficient model routing creates more complexity in the fabric that carries it.
The companies achieving the best results are not only the ones with the most powerful raw compute. They are the ones who treat the network as a first-class design decision alongside compute, software, topology, and communication libraries.
Designing a Network Built for AI
Getting the most out of an AI cluster requires more than raw compute. It requires a network architecture designed for the workload and a software stack that can optimize communication across the cluster.
First, the fabric needs to be designed for AI traffic patterns. Traditional data center networks were built for general-purpose traffic, not for thousands of GPUs exchanging synchronized data at extremely high speed. AI fabrics need high bandwidth, low latency, and predictable performance under heavy load.
Second, the network needs to handle congestion before it results in wasted GPU cycles. This may include fabric-level scheduling, endpoint-level scheduling, adaptive routing, load balancing, and topology-aware placement. The goal is to prevent hotspots and make better use of available paths across the cluster.
Third, the communication stack must be optimized together with the network. Collective communication libraries play a major role in training performance. Tuning them for the specific accelerator, topology, and workload can improve how data moves within a node and across nodes.
Fourth, the cluster should be treated as one system, not as separate compute, network, and software domains. The best results come when infrastructure teams understand the relationship between the model architecture, training framework, interconnect, and physical network.
Solving the problem is not just about buying faster switches or adding more links. It is about designing an AI fabric that matches the model’s behavior and the underlying hardware.
It Comes Down to Compute Plus Network
The broader lesson is consistent across all of this. The most important infrastructure decision is not how much compute you deploy. It is how well all the pieces connect.
The compute is there. The models are there. And while both continue to evolve, what ultimately determines whether that multi-billion-dollar investment delivers its full potential is how well the underlying AI network fabric is designed and optimized.