Written by Apoorva Joshi — Staff AI Developer Advocaite at MongoDB.
As enterprises integrate AI into their most important internal and customer-facing workflows, they are inadvertently creating new attack surfaces and taking on new categories of risk and liability. In July 2025, a Replit AI agent deleted a live production database during an active code freeze, despite repeated instructions not to make changes. In April 2026, a Cursor agent destroyed an entire production database, including backups, by autonomously locating and using an API token with broad permissions that it shouldn’t have had access to. In each case, the agent did exactly what it was technically capable of doing, because nothing stopped it. These aren’t model failures, but governance failures. As AI moves from experimentation to production, governance is the difference between scaling with confidence and stalling.
Unlike earlier-generation AI systems that surface predictions or recommendations for a human to act on, AI agents act directly, making purchases, sending communications, and modifying data and systems. And because they reason dynamically rather than executing a fixed sequence of steps, their behavior is shaped by whatever they encounter at runtime: the tools available, the data in context, and their own judgment about how to accomplish the goal. Frameworks like OpenClaw push this further, with always-available, proactive agents connected to your messaging surfaces and tools.
For AI agents, governance comes down to three things: defining what data and systems an agent can access and with what level of permissions, defining which actions an agent can take autonomously and which require human approval, monitoring agent behavior in real time to detect when it drifts outside acceptable boundaries, and maintaining audit trails detailed enough to reconstruct the full chain of actions when something goes wrong. Building a governance framework for AI agents is ultimately an organizational problem that requires alignment across legal, compliance, security, and engineering teams. That’s out of scope for this article. Here, we’ll assume your organization has AI governance policies in place and focus on how to enforce them within the AI systems you build.
The advent of AI coding tools has made it easier than ever to go from 0 to 1. But this has also increased the tendency to jump straight to building without thinking through what you’re solving for. Your organization’s AI governance policy will likely specify regulatory frameworks you must comply with, data access and residency requirements, and decisions that require human approval. All of these have direct implications for how you design your system.
For example, let’s say you’re building an AI-powered claims review system for a health insurance provider. HIPAA [2] requires that Protected Health Information (PHI) be processed and stored only on approved, compliant infrastructure and used for the specific purpose for which it was collected. In practice, this means you can’t send PHI to a model provider that hasn’t signed a hasn’t signed a Business Associate Agreement (BAA) with your organization, you can’t run inference outside your approved cloud environment, and you can’t use patient data to fine-tune or improve models without explicit authorization. Your governance policy may also require that high-stakes decisions, such as claim denials, are reviewed by a human before a final decision is made, which means human-in-the-loop checkpoints need to be designed into the system from the start, not added as an afterthought. Not identifying these constraints upfront can expose your organization to regulatory penalties and legal liability. Retrofitting governance into a system that wasn’t designed for it is an expensive way to learn that lesson.
Guardrails are checks that run at specific points in an agent’s execution pipeline to enforce the boundaries defined by your governance policy.
Guardrails can be deterministic, for example, a check that confirms that the file the agent wants to read is in an approved directory. Or they can be model-based, meaning they use LLMs or other machine learning models to evaluate cases that can’t be reduced to a simple rule. For example, an LLM with a prompt that helps it evaluate whether an input looks like a prompt injection attempt, or a toxicity classifier that scores whether the agent’s response contains harmful content.
Guardrails should run at three layers: input, execution, and output, since each layer catches a different class of failure. Input guardrails should check what enters the system, catching prompt-injection attempts, blocking requests containing sensitive data, and rejecting queries that fall outside the agent’s intended scope. Execution guardrails should run before a tool is invoked to verify that the action is permitted under current policies. Output guardrails run after the model responds, but before the response is returned to the user or passed downstream. They check for hallucinated, malformed, and harmful outputs.
For our healthcare claims review example from earlier, a model-based input guardrail could evaluate whether the request appears to be an attempt to manipulate the agent’s instructions rather than a legitimate review query. An execution guardrail could block automated final decisions for denials, thereby enforcing the human-in-the-loop requirement established in your governance policy. An output guardrail could check that the final recommendation is accompanied by citations for the clinical guidelines and coverage policies that informed it. One final point: a guardrail you haven’t tested is an assumption. Before deploying your agents, run adversarial inputs through them to ensure your guardrails are working as expected.
Guardrails catch failures within the boundaries of what your agent can do. Deny-by-default is how you set those boundaries in the first place. The principle stems from zero trust, a well-established security model built on the idea that no user, device, or system should be trusted by default. Agents are a new type of system, so the same principle should apply to them, too.
Deny-by-default is an additive model in which permissions are built up from nothing, and every capability must be explicitly granted. This introduces oversight into the process by design, since a human is explicitly provisioning access to data and systems in a way that respects the organization’s governance policies. A related principle is least privilege: when granting access, scope it to the minimum the task requires, specific collections rather than entire databases, specific API endpoints rather than full services. In practice, both are implemented through role-based access control (RBAC), in which you define explicit roles tied to specific tasks, and each agent is granted only the roles it needs. Together, these three principles form the foundation of a sound permission model for AI agents.
Deny-by-default should also govern which actions an agent can execute autonomously. For irreversible, high-stakes operations such as deleting records, sending external communications, triggering payments, and issuing decisions, the right pattern is to deny by default and require human-in-the-loop approval before executing the action.
Both the Replit and Cursor incidents were failures of the same three principles: deny-by-default, least privilege, and human-in-the-loop, and an absence of execution guardrails to enforce them. In both cases, agents had access to systems and capabilities well beyond what their tasks required. In the Cursor case, that’s how the agent found an API token in an unrelated file. The token itself violated the principle of least privilege. In both cases, human approval for destructive actions was a prompt instruction rather than a system-level control, exactly the kind of enforcement an execution guardrail is designed to provide.
Auditability is table stakes for governance. When things go wrong, you need to be able to reconstruct exactly what happened, why it happened, and who was responsible. From a regulatory standpoint, auditability is no longer optional. The EU AI Act [3] now requires organizations deploying AI in high-stakes contexts such as healthcare, finance, and critical infrastructure to maintain queryable records of AI-driven decisions. In financial services, FINRA [4] treats missing traces as a books-and-records violation. HIPAA requires that compliance documentation be retained for at least six years.
The foundation of a meaningful audit trail for AI agents is agent identity. Access and audit logs need to distinguish between humans, AI agents acting autonomously, and agents acting on a human’s behalf. Without agent identity established at the system level, you can’t answer the questions an audit actually demands: was this decision made by a human or an agent? Was the agent acting within its authorized scope? Which agent, under what authorization?
Establishing agent identity means treating each agent as a distinct principal in your identity and access management (IAM) systems, just as you would treat a human user. In practice, this means each agent has its own unique identifier, its own scoped credentials, and its own entry in your access and audit logs. Two patterns are worth knowing here: For agents acting on a user’s behalf, OAuth’s On-Behalf-Of [5] authorization flow proposes a way to make delegation explicit and traceable, so the log reflects not just that an agent acted, but on whose behalf and within what authorization scope. For autonomous agents, SPIFFE [6] defines a standard for issuing each agent a short-lived, cryptographically verifiable identity tied to its runtime environment, so that system interactions can be attributed to a specific, verified entity rather than a shared or static credential.
When building your next AI agent, start with these questions so governance requirements shape the system design before implementation starts.
What are your governance constraints? Have you read through your organization’s governance policies and distilled them into specific design requirements? If you’re building in a regulated industry, can you name the frameworks you’re subject to, the data you can and can’t send to your model provider, and the decisions that require human approval before an agent can act?
What are your system’s guardrails? What can this agent do, what should it never do regardless of instructions, and what decisions require human intervention? Does your system design capture these requirements?
What does deny-by-default look like for this system? What are the systems that this agent will connect to? What level of access should it have to these systems? Before you connect your agent to any system, data store, or tool, ask whether that access is explicitly required for the tasks the agent needs to perform.
**How will you make this system transparent and auditable? **If a regulator asked you to produce a record of every decision your agent made, who authorized it, and whether a human was in the loop, could you? How and where will your system store access and audit logs, and how will it distinguish between actions taken by humans and agents?
Governance by Design: Four Principles for Building Safe, Compliant AI Agents was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.