Many IT leaders looking to capitalize on the promise of agentic AI still struggle with a basic step — building out their agentic architecture — even as they roll out dozens or hundreds of agents.
AI agent deployment is expected to skyrocket over the next year, with IDC predicting a tenfold increase in agent use by large enterprises by the end of this year. But in some cases, CIOs haven’t focused on the building blocks needed to run a huge team of agents, experts say.
Many CIOs have underestimated the infrastructure work required to get agents to scale responsibly, says Hilary Packer, EVP and head of enterprise data and AI at American Express.
“There’s a tendency to focus on what an agent can do and then move quickly to deployment before the underlying infrastructure is in place,” she says. “It is essential to first build the enterprise capabilities that allow agents to operate consistently across systems, data sources, and workflows.”
IT leaders must also set up well-governed data foundations, but many companies still operate with fragmented systems, inconsistent data definitions, and siloed ownership structures, Packer adds.
“Before deploying increasingly sophisticated models and agents, organizations need confidence that the underlying data is accurate, accessible, and fit for purpose,” she says. “That requires investments in governance, lineage, and modern data infrastructure that allows information to move efficiently across the enterprise.”
Packer sees agentic AI as much a governance and operating model challenge as a technology challenge. In some cases, IT teams are reinventing the wheel with each agent pilot, she suggests.
“As organizations experiment, teams often end up rebuilding the same capabilities repeatedly — for example, identity management, access controls, monitoring, and observability,” she says. “That may be sufficient for isolated pilots, but it becomes difficult to manage and scale across the enterprise.”
CIOs need to take a wholistic approach to agentic architecture. While some IT leaders think of architecture as the AI model powering agents, it goes much deeper.
Agentic architecture includes several elements, and in some cases, CIOs have underinvested in some functionality, says Saurabh Pitkar, director of product management for agentic commerce at Dell Technologies.
Agentic architecture, Pitkar says, includes an orchestrator, the brain that controls subagents; subagents themselves, which have specialized functions; APIs and other tools; memory to maintain context of an agent session and overall behavior over a longer period; and guardrails to set boundaries for agents
Many organizations are now building agentic architecture, but those still struggling have failed to make the right investments in areas such as building memory and creating MCP tool standardizations, Pitkar says.
A lack of data access and integration can be a huge barrier to agent deployments despite heavy investments in data modernization, he adds.
In addition, IT leaders struggling to deploy agents have often failed to manage organizational changes to account for a faster, agent-driven delivery cycle, he says. Adoption can slow down considerably if teams aren’t ready to engage with agent-speed outputs.
“AI does not care about org structure when it comes to accessing data,” he says.
While the elements of agentic architecture are fairly standard, CIOs may need to focus more on different functionality depending on the use case, according to Pitkar. The devil is in the details. “Customer support may need higher investments in dynamic intent mappings, memory, effective UX as these are emotionally charged conversations with human users who need a problem solved to continue their job,” Pitkar adds. “Agentic commerce may focus on deterministic APIs with machine readable data, guardrails, and compliance to securely process transactions with real money.”
Another way to look at agentic architecture is to think about four layers:
Most organizations have these four functions happening in other enterprise software packages, but they are built for human-paced orchestration instead of machine-speed agents, says Shibani Ahuja, SVP for enterprise IT strategy at Salesforce.
“Agentic architecture is the enterprise foundation that allows AI agents to autonomously reason and act — not just respond — in a way that’s governed and secure,” she says. “It’s the difference between an AI that drafts an email and an AI that closes tickets, triggers payments, and updates records, without waiting for a human to pass the baton.”
Ahuja sees organizations struggling to even define agentic architecture, much less deploy it. Throughout 2025, she heard CIOs describing how they implemented AI on a scale.
“If you listened carefully, one was describing predictive AI, one generative AI, and one an agentic workflow that was really just a sophisticated chatbot,” she says. “We were using the same word for fundamentally different things. When the ROI didn’t materialize, organizations didn’t know what had actually failed.”
Agent deployment struggles, meanwhile, have been less about capability and more about where organizations focus their energy, Ahuja says. Many IT leaders focus on the agent, including the use case, the prompt, and the model, but it’s just as important to ask whether the organization’s architecture actually supports its agentic goals, she adds.
“Can agents access real-time, governed data across your entire business?” she says. “Can they act across systems with minimal human input to reconcile inconsistencies? In most enterprises, the honest answer is not yet, and no amount of prompt tuning fixes that.”
It’s now trivial to set up an agent, with most organizations able to do it in days, if not hours. But the work doesn’t stop there, says Adam Field, chief AI officer at workflow automation provider Tungsten Automation.
Good agentic architecture allows agents to operate reliably inside real business processes, not just in isolation, he says. For the past 30 years, enterprise systems were built for humans to navigate UIs, but agents work silently on the inside by calling APIs and requiring action-level permissions rather than login access.
The underestimated piece of agentic architecture is the governance controls that define what agents can do autonomously and when they must stop and hand off control to a human, Field says.
“Deploying an agent on a discrete task is straightforward,” he adds. “The hard part is that businesses don’t run on discrete tasks.”
Instead, enterprises operate on end-to-end, regulated processes with exceptions, dependencies, compliance requirements, and humans in the loop at specific moments, he notes.
“Designing agentic architecture that works across an entire process, not just a single step, is a fundamentally different challenge,” Field adds. “Traditional software fails obviously. Agents fail silently, confidently, at scale.”