AI workload placement has quietly become one of the biggest challenges that our customers face in their deployments. It’s not just about getting a project off the ground anymore; it’s about where that workload lives. That one decision ends up dictating so much: your costs, security, control, power, and latency.
To better understand how organizations are making decisions on where their AI workloads should run, we worked with Omdia to survey over 1,200 infrastructure leaders. The results reveal trends that span across industries and geographies.
Why 94% of infrastructure leaders are rethinking their AI strategy #
The most striking takeaway from the research was that nearly every organization surveyed (94%) now reports regrets regarding their initial AI infrastructure strategy.
The research suggests the issue was less about AI technology failing and more about early decisions moving faster than teams could fully understand the economics, security requirements, and workload realities. While 96% of organizations shared that they have already adopted a hybrid approach—across cloud, on-premises data centers, and edge infrastructure—getting that mix right is proving to be a significant challenge.
The real mistake: skipping the workload analysis #
When we asked these infrastructure leaders what they would do differently, their answers were not only about changing platforms. Instead, they reflected on the strategic gaps they only recognized in hindsight.
- 34% of respondents wanted more rigorous cost analysis before the first deployment.
- 33% said they would have invested in their own infrastructure earlier.
- 28% felt they should have pushed back harder on the pace of
Where do AI workloads belong: cloud, on-premises, or both? #
Once you get specific about your AI goals, the debate over where AI should live stops being a general argument and becomes a practical conversation about optimization. In fact, 54% of organizations cite workload-specific optimization as the top reason for adopting a hybrid strategy.
That’s why we’re seeing a shift. Organizations are splitting their strategies based on priorities and requirements for a given workload:
On-premises for control: Custom models built on proprietary data (like financial algorithms or manufacturing specs) are staying (or moving) on-premises. This helps ensure data sovereignty, tighter security, and a predictable cost structure.Cloud for scale: Less sensitive, standardized AI functions may be better suited to cloud or hybrid approaches when organizations need faster deployment and flexible scale.
What this looks like in practice: more than half of organizations adopting hybrid AI (54%) say their top objective is optimizing workloads based on their specific needs. As one manufacturing CIO surveyed put it: “Anything that needs a real-time decision stays on-premises, but everything else can go to the cloud.”
That kind of deliberate split, made early and revisited often, is what keeps leaders ahead of the regret that so many others are now confronting.
How network capacity is expected to become the next AI bottleneck #
Making the network an afterthought is a common hurdle in scaling AI. Even as organizations scale server capacity for AI, the network is emerging as an even more immediate bottleneck.** **This is a growing concern for infrastructure leaders, with 67% of the respondents expecting their network to hit its limits due to the deluge of AI traffic within 12 months.
For practitioners and data ops leaders, we think this can be a critical failure point. If your network drops packets, your expensive GPUs sit idle waiting for data. Idle GPUs mean delayed Job Completion Times (JCT) and wasted investment. As we expand into distributed AI and edge computing, the coordination problem multiplies. Workload placement decisions pay off only if the underlying infrastructure—compute, network, and security scale as one integrated system—with unified observability and a consistent operating model, collectively driving performance.
Treat AI infrastructure as a continuous strategy, not a one-time bet #
If you’re currently evaluating your AI strategy, here’s my advice: Audit your workload placement: Optimize AI workloads by use case, balancing total cost, control, performance, and speed to deploy across cloud, on-premises, and edge environments.Treat network capacity as a prerequisite: Don’t wait for your network to hit its limit before you plan for scale. It is the foundation of your AI performance.Build your strategy as a system: Your data center and edge strategies should move in lockstep, not sequentially.
The AI infrastructure bet isn’t a one-time decision. Getting it right means treating workload placement as a continuous, deliberate process, revisiting your strategy as your workloads mature.
See the full data behind this shift in AI infrastructure strategy #
Download the complete Omdia research report to see exactly what 1,201 IT decision makers revealed about AI infrastructure regrets, workload placement, and the decisions they would revisit. And to learn more about how Cisco is delivering unified architecture for data center networking environments, explore Cisco Nexus One.