Large Australian banks have started implementing agentic AI at scale this year. The same is happening across the country’s larger enterprises. These are not pilots, but production systems that are the tip of a global trend. Research company Gartner expects 40% of enterprises to embed AI agents in applications by the end of 2026, up from less than 5% in 2025 — an eightfold jump in 12 months.
Most governance teams at enterprises are responding to these changes by building playbooks as they always have: through committees, policies, approval gates and periodic audits. It’s a model that assumes humans review most decisions and that governance is a central function that sets and enforces all the rules. None of those assumptions holds for AI agents. The playbook most organisations are drafting at this moment is already obsolete before it is even finished.
In our view, three central changes must happen together for AI governance to work in environments where agents are used at scale:
“You cannot govern what you cannot measure” sounds like a platitude, but most enterprises do not yet know how to measure AI agents’ actions. For instance, what counts as normal agent behaviour? What telemetry tells you that an agent is drifting from its original scope? What does an incident look like when it is not a single breach but four hundred micro-decisions, each individually defensible, that add up to an outcome you would never have approved or anticipated?
Businesses are still working this out. In the meantime, they are building and deploying agents without the capacity to see or understand in detail what they are doing.
A 2026 Gravitee survey found that only 24.4% of organisations report having full visibility into how AI agents communicate with one another. Almost nine in ten (88%) have reported “confirmed or suspected” agent security incidents in the past year. Despite that, 82% of executives say they are confident that existing policies protect against unauthorised agent actions.
These numbers indicate a widening gap between confidence and risk whenever AI agents are deployed. In my experience working with enterprise clients across the region, this gap is not a matter of negligence. Most governance teams are doing what they have always done well. The problem is that the systems they are now responsible for are behaving in ways their existing tools were never designed to detect.
Visibility matters because policy, control frameworks and ethics boards are all scaffolding that collapses without behavioural data that can be analysed and acted upon. You cannot write a meaningful policy for a system whose normal actions and performance you have never characterised or audit an agent whose decisions you cannot investigate or understand.
In this context, observability means instrumentation, baselines, anomaly detection on agent behaviour and telemetry that humans can interpret.
Once you can see and understand what agents are doing, the next problem is how to manage them at scale. When the average enterprise runs 12 agents, as Salesforce’s 2026 Connectivity Report suggests, human oversight is still feasible. When leading deployments are already running into the hundreds — IQVIA has deployed more than 150 agents, for example — that approach stops working.
We don’t necessarily need a completely new security framework, but the updated model must allow companies to operate non-human interactions at scale with confidence. And the only way to ensure this is to use AIs to govern other AIs, because it is not economical to rely solely on humans to do the work.
This agentic AI governance model needs to monitor agentic behaviour and respond within milliseconds when needed. It must provide situational awareness and key insights to support informed decision-making.
Humans are responsible for establishing the parameters and guardrails, but they only intervene on demand and spend most of their focus on continuously improving their AI governance capabilities and the governance agents.
If governance must be observed continuously at machine speed and at a high level of complexity, no single function can do it all. That is why another essential change is organisational, with accountability distributed by design. Today, most businesses use a centralised model that no longer works. Legal owns policy; security focuses on runtime monitoring and response; developers build controls into the agents themselves. The problem is that each function is limited in its own way.
Security can monitor telemetry data, for instance, but lacks insight into what each agent is supposed to do and therefore cannot develop customised anomaly-detection controls.
Closing the gaps in the overall systems requires understanding a specific approach to building agents. Developers cannot simply ship unmanaged agents that operate in stealth mode. They need to send key metrics to a centralised AI governance layer. To enable this layer, a clear shared responsibility model between the developer and governance functions must be defined. Developers are required to implement reporting hooks that generate data to create key metrics and task-specific insights and detect anomalies across clearly defined governance domains.
The centralised AI governance layer analyses this incoming data from the agents and provides situation awareness across all deployed AI agents. Guardrails that are baked into AI agents can’t be trusted, as they have proven to be vulnerable to prompt injection attacks. That is why an independent AI-powered governance layer is required to supervise all agent behaviour and provide insights and key metrics to key stakeholders.
Distributed accountability like this is hard to set up. It requires an understanding of the reasons behind the changes and an agreement between many stakeholders on how the new model must operate and where different responsibilities lie. It also needs a shared language across functions that have not historically worked together at this pace and clarity about who decides what when something goes wrong. But it is the only model that survives an environment with AI agents deployed at scale.
One way to think about these changes is to reflect on cloud adoption over time. That experience showed us that investing in governance and assurance early reduced risks and created a competitive advantage for the businesses that understood why they should do it. The same dynamic is playing out with AI agents, only faster, more distributed and very likely at a much larger scale.
Managing security, compliance, privacy, responsible AI, quality and cost in an agentic world at machine speed hasn’t been done before. Vendors help innovate on the customer’s behalf and can offer building blocks for this governance layer. But organisations also need to consider creating AI-powered custom capabilities to fill their processes and observability gaps. AI governance teams, therefore, require engineering capabilities and an agentic development environment that is tightly integrated with out-of-the-box AI security and compliance solutions.
What I see across the market right now is that the professionals responsible for governance have the expertise and experience to lead this shift. What they need is the confidence to rethink their operating model and recognise that the centralised control they are accustomed to will not scale for agentic environments. Good governance practices that consider people, processes and technology have always paid off in the long run. Now is the time to define the individual North Star for your AI governance layer, because retrofitting these capabilities will carry high risk and cost.
This article is published as part of the Foundry Expert Contributor Network.Want to join?