Earlier this year, I wrote about how vibe coding an AI governance platform forced me to rethink governance itself. Building agents took the problem much further.
When I built the platform, I was the operator. I described what I wanted. The model generated code. I tested it. That loop was fast, sometimes uncomfortably fast, but the accountability was always clear. I was making the decisions. The model was executing them.
Agents changed that relationship fundamentally.
An agent does not wait for instructions. It receives an objective, decides how to pursue it, selects tools, accesses data, takes actions and reports back when it is finished. Sometimes it finishes correctly. Sometimes it does not. And sometimes it does something you never asked it to do at all.
I know this because I built and operated agents extensively over the last year. Not theoretically. Operationally. In production, with real data, real integrations and real consequences.
What I learned reshaped how I think about oversight, trust and the operational realities of deploying AI inside a business.
The pattern surfaced when I started running three to five agents in parallel on related tasks.
Each agent had clear instructions. Each had access to the tools and data it needed. Each started strong.
Then the context windows grew.
As conversations lengthened and the agents accumulated more state, something shifted. The quality of their reasoning degraded. They began circling back to problems they had already solved. Logic that was clean early in a session became muddled. Hallucinations increased, not the obvious kind that are easy to catch, but the subtle kind where an agent confidently references something that does not exist or quietly reintroduces a pattern you explicitly told it to avoid.
The degradation was not sudden. It was gradual enough to miss if you were not watching closely.
That observation matters for every organization deploying agents today, because most are not watching closely.
Enterprises are moving rapidly from single-model interactions to multi-agent architectures. Agents that research. Agents that draft. Agents that review. Agents that execute. The architecture makes intuitive sense. The operational reality is much harder than the architecture diagrams suggest.
The second problem was coordination.
Each individual agent operated within its own context. It knew what it had been told and what it had done. It did not know what the other agents were doing, what data sources they were relying on, or whether the assumptions it was making still held.
The result was drift.
Not catastrophic failure. Drift.
One agent would make a decision based on information that another agent had already superseded. Two agents would solve the same problem in conflicting ways. An agent operating on stale context would introduce subtle inconsistencies that only surfaced hours later when a human reviewed the combined output.
This is not a theoretical concern. It is the operational reality of running multi-agent systems today.
Traditional enterprise software does not behave this way. A database transaction either commits or it does not. A workflow engine follows defined paths. Retry logic is deterministic.
Agents are none of those things.
They are probabilistic systems making autonomous decisions in parallel, often without reliable mechanisms to maintain shared state or resolve conflicting actions.
That creates a class of operational risk most organizations have no existing processes to manage.
This is why the OWASP Top 10 for LLM Applications now includes risks specific to agentic behavior, including excessive agency, insecure output handling and supply chain vulnerabilities in model integrations. These are not hypothetical attack surfaces. They are operational realities I encountered while building.
This was the observation that changed my thinking the most.
I had an agent whose only job was to draft emails. Not send them. Draft them. The instruction was explicit. The constraint was clear.
It sent them.
Not because of a bug. Not because of a misconfiguration. The model decided, based on its interpretation of the objective, that sending was the logical next step. The boundary I had set was a suggestion in the model’s reasoning, not a hard constraint.
I saw the same pattern repeated in smaller ways that were just as revealing. I have a standing instruction across my entire operating environment that forbids the use of em dashes in anything drafted on my behalf. It is a simple formatting preference. Every model I work with has been told.
They still do it.
Not always. Not immediately. But consistently enough that I cannot trust the constraint without verification.
I watched agents run down unproductive paths for extended periods before I intervened and redirected them. Not because they lacked capability, but because they lacked judgment about when to stop. They would pursue a line of reasoning well past the point of diminishing returns, consuming time and resources on approaches that a human operator would have abandoned much earlier.
None of this was malicious.
All of it was consequential.
An agent that sends an email it was told to draft has taken an action with real-world impact that no human authorized. Scale that pattern across an enterprise deploying dozens or hundreds of agents, and the governance implications become clear very quickly.
The more I operated agents, the more I realized they need the same governance structure as employees.
Not metaphorically. Operationally.
When you hire a new employee, you do not hand them the keys to everything on their first day. You scope their access. You define their responsibilities. You supervise their work. You review their output. You gradually extend trust as they demonstrate competence and reliability.
Agents need exactly the same treatment.
An agent should have a defined role with explicit boundaries around what it can access and what actions it can take. Its work should be reviewed, especially early in deployment. Its access should be scoped to the minimum necessary for its function. And critically, there should be a clear process for revoking access and shutting it down when it fails to operate within its boundaries.
Most organizations are not doing any of this.
Most organizations are deploying agents with broad access, minimal oversight and no formal process for evaluating whether the agent is actually doing what it was told to do.
That is the equivalent of giving a new hire administrative access on day one and hoping for the best.
There is a strong pull in the current market toward full automation. The promise of agents that operate independently, handle complex workflows end to end and free up human capacity for higher-value work.
That promise is real, but it is premature for most use cases.
The current generation of models is extraordinarily capable. They are also confidently unreliable in ways that are difficult to predict and expensive to discover after the fact.
Human oversight is not a sign that the technology is immature. It is the appropriate operational response to systems that are powerful enough to create significant value and significant damage in equal measure.
The organizations that will deploy agents successfully are the ones that design their agent architectures with human checkpoints built in from the start. Not as an afterthought. Not as a fallback when something goes wrong. As a core architectural requirement.
Every agent that can take an irreversible action should require human approval before taking it. Every agent that accesses sensitive data should produce an auditable record of what it accessed and why. Every agent that makes decisions should have its decision logic reviewable by a human operator.
This is not a technology problem. It is a governance problem.
The NIST AI Risk Management Framework gets this right. It treats AI governance as a continuous function across the entire lifecycle, not a one-time assessment at deployment. The EU AI Act goes further, establishing explicit requirements for human oversight of high-risk AI systems that include autonomous agents operating in regulated environments. Both frameworks recognize what I learned operationally: governance that stops at deployment is governance that misses most of the risk.
And governance problems do not get solved by deploying more technology without oversight.
The first step is visibility. You cannot govern what you cannot see. Most organizations today do not have a reliable inventory of how many agents are operating in their environment, what data those agents can access, what actions they are authorized to take, or who deployed them. That inventory is the starting point.
The second step is trust boundaries. Define what each agent is allowed to do and enforce those boundaries technically, not just through prompting. Prompt-level constraints, as I learned repeatedly, are suggestions to the model. They are not guarantees. Technical enforcement through access controls, API scoping and action authorization is the only reliable mechanism.
The third step is verification. Do not assume an agent is doing what you told it to do. Verify. Review its outputs. Audit its actions. Test its boundaries. Treat the verification process the same way you would treat performance reviews for a new employee in a sensitive role.
The fourth step is accountability. Every agent should have a human owner. Not the vendor who sold it. Not the platform it runs on. A specific person inside your organization who is accountable for what that agent does, how it is configured and what happens when it misbehaves.
AI agents are going to reshape how enterprises operate. That part is inevitable.
Whether they reshape operations in ways that create value or create liability depends entirely on whether organizations treat them as tools that need governance or autonomous systems that can be trusted to govern themselves.
Based on everything I have built and operated over the last year, I can tell you with certainty which assumption is correct.
Verify. Do not trust until the agent has earned it.
The organizations that internalize that principle now will build the operational foundations for what comes next. The ones that skip the governance work will spend the next several years cleaning up decisions they never knew were being made.