Most AI roadmaps rest on a quiet bet: that the labs will eventually ship a model safe and aligned enough to simply trust in production. Better training, better guardrails, one more version, and the thing behaves.
Here is the flaw in that bet. Even a perfectly aligned model cannot tell you who used it, on what data, under whose policy, or hand you a record you could show an auditor. Those are not facts about how the model behaves.
They are facts about how it was deployed, and they live entirely outside the weights. A better model answers a different question than the one regulators, auditors, and security teams are actually asking, and the gap between those two questions is where the real risk now sits.
Security engineering named this trap fifty years ago. A 1972 US Air Force study defined the reference monitor, the component that decides whether an action is allowed, and set three conditions for trusting it: it must be tamperproof, invoked on every access, and small enough to be completely verified.
A modern frontier model is none of the three. Alignment tries to make the model enforce the rules meant to constrain it, from the inside, and a system cannot be the thing that governs itself. Governance has to sit around the model, not in it.
Safety is not the same as governance
The confusion hides in one word. Safety and alignment ask whether a model tends to behave well. That is a disposition, it lives in the weights, and the labs have gotten genuinely good at shaping it. Governance asks something different: who used which model, on what data, under whose policy, and with what auditable record.
That is a property of a specific deployment. Training can shape the first. It cannot, by construction, supply the second. Treating them as one problem is the category error underneath most AI risk conversations.
The problem is bigger than the model, and it is growing
This is not a niche concern for the labs. It lands on everyone who deploys AI, and it is expanding for two reasons.
First, models are gaining agency. When a model only produced text, a bad interaction meant a bad answer. Now models take actions through tools and protocols like MCP, and every tool call is a fresh decision about who is acting, on what data, under whose authority. Those decisions multiply with every task and every user.
OWASP’s 2026 tracking finds that most of the agentic projects it follows are coding agents, and that prompt injection maps to six of the ten risks on its agentic top-ten list. The ungoverned surface grows as fast as adoption.
Second, the accountability is now legal. California’s SB 53 and the European Union’s rules for general-purpose AI put obligations on the labs, but the EU AI Act also places separate duties on deployers, and data-protection law already holds organizations responsible for how personal data is used.
When a regulator asks what your AI did, “we used an aligned model” is not an answer, and a vendor’s safety report will not tell them who inside your company sent which data to which model last Tuesday.
Public benchmarks do not help either: StrongREJECT, HarmBench, and AgentHarm grade the model on generic prompts, not your users, your data, or your policy on a given day. A high score is a property of the model, not an audit trail. So the scope is wide and concrete: every organization that deploys AI, a surface that grows with every new agent, and a liability that is now enforceable. None of it is a model-quality problem.
Why aligning the model cannot fix it
Go back to the 1972 test, because it shows exactly where the fix cannot live.
A model is not tamperproof. Prompt injection, named in 2022, works because instructions and data ride the same channel of text, so a document the model reads can overwrite the instruction it was given.
Four years on it is unsolved: OWASP says it is unclear whether fool-proof prevention exists, and a 2026 paper argues agents may always be vulnerable. A control you can rewrite with the input it is inspecting is not a control.
A model is not reliably invoked as a gate. Give it tools and it becomes a textbook confused deputy, a 1988 term for a trusted program that an untrusted caller tricks into misusing its authority, because it cannot tell an instruction from the data it is reading.
A model is not verifiable. You cannot inspect billions of opaque parameters the way you can audit a small enforcement kernel.
Fail all three tests and the conclusion is not “align harder.” It is that the model cannot be the enforcement mechanism, so the enforcement has to live somewhere else.
The fix is a layer around the model
If the model cannot govern itself, the answer is not a better model but a thin control layer around it, one that treats the model as an untrusted component and mediates everything it tries to do. This is not new architecture to invent. It is the 1972 reference monitor rebuilt for an agent, and here is how it works.
Picture a request moving through it. The model reads its input and proposes an action: call this API, write this file, send this message.
That proposal is never trusted on its own. It leaves the model as a request, not a command, and passes to a small policy engine outside the model, sitting in the path between the agent and the resource it wants to reach, the way zero-trust architecture separates the point that decides whether an action is allowed from the point that carries it out.
The engine checks the action against two things the weights never had: your actual policy, and the narrow set of capabilities this agent was granted for this task.
If the action clears both, an enforcement point performs it; if not, it is refused. Either way, who acted, through which model, on what data, under which policy, and what was decided is written to a log. That one flow restores the three properties the model could not provide. It is tamperproof, because the policy lives in a small external engine rather than in text the model can be argued out of. It is always invoked, because nothing reaches a real resource without passing through it.
And it is verifiable, because an enforcement point small enough to read is one you can actually trust, in a way billions of opaque weights never can be. This is why the strongest research defenses already work this way: CaMeL, from Google DeepMind and ETH Zurich, gets provable security by containing the model exactly like this, and it holds even when the underlying model stays vulnerable. The guarantee lives in the architecture, not the weights.
None of this makes the model safe, and that is the point. It makes the surrounding system accountable, which is a smaller and far more achievable goal, and it is the one the deployment problem actually demands.
The right layer
This is not an argument against alignment. Alignment is necessary, and it is improving. It is simply the wrong layer for the question that regulators, auditors, and users are actually asking. The answer to that question has been on the shelf since 1972, and the industry keeps re-deriving it the expensive way.
The next model will be safer than this one. It still will not know who used it, on what data, or under whose policy. Closing that gap is the work, and it is the story.
Get the TNW newsletter #
Get the most important tech news in your inbox each week.
Expertise from selected TNW Council members, admitted through an application and review process to a fee-based program. Opinions expressed by the authors are their own.