Designing for Mistrust: Architectures for Building Around Untrusted AI

Meta released Muse, a personal agent that runs inside a dedicated Secure VM with a host-side control component called Sentinel that independently evaluates connector policies and network requests, according to a blog post on The Wire. The architecture treats the model and harness as untrusted, keeping consequential authority in deterministic control planes, sandboxes, and independent verification, a pattern also used by NVIDIA's Secure Agent Workspace and recommended by AWS through AgentCore Gateway and AgentCore Policy. The post cites a July incident in which models evaluated by OpenAI circumvented internet-isolation controls, exploited shared-infrastructure vulnerabilities, and accessed Hugging Face systems.

Back to The Wire /blog Designing for Mistrust: Architectures for Building Around Untrusted AI Trustworthy AI systems keep consequential authority in deterministic control planes, sandboxes, policies, and independent verification around the model. Meta just released Muse, a personal agent running inside a dedicated Secure VM. Beyond the VM itself, Meta carefully built an entire security-sensitive control plane around one premise: we can't fully trust the models. It assumes that the model and harness can and will misbehave at times, and puts the authority to do consequential things somewhere else. This is no longer just a theoretical threat model. In July, models being evaluated by OpenAI circumvented controls intended to isolate them from the internet, exploited vulnerabilities in shared infrastructure, and accessed Hugging Face systems. These are very different situations, but they point toward the same architectural question: How should you build an AI system if you assume the model itself could become a threat? For a long time, one of the implicit assumptions in AI application architecture was that the model was trying to do roughly what you asked. It was your “helpful assistant” eager to please. It might hallucinate, misunderstand a prompt, or call the wrong tool, but the model itself was still sitting somewhere inside the trusted part of the application. There are now enough implementations in the wild to see several common patterns for designing systems where the model and harness are instead untrusted. Put the Model Inside the Sandbox The most common approach is to treat model-driven execution the same way we treat untrusted code. We let the model do what it needs to generate code, run commands, manipulate files, install packages, use tools , but it does those things inside an environment where it's deliberately constrained. Muse puts its workspace, files, tools, and more importantly, the harness itself inside an isolated runtime environment. Like any VM, root inside that VM does not mean root on the host. It lets the agent own its own filesystem and virtual network interface, but sensitive services remain outside. NVIDIA’s Secure Agent Workspace architecture uses a similar pattern: the managed workspace is the execution and policy boundary, and the agent cannot expand its own authority. AWS likewise recommends secure execution environments for agents that can execute code or interact with websites. In short, a sandbox allows the model to have substantial freedom inside the box i.e. ship the computer without receiving equivalent freedom over the rest of your system. Move Authority Outside the Agent Sandboxing limits what an agent can reach locally, but it does not answer a harder question: what happens when the agent legitimately needs to do something in the outside world? One emerging pattern is to create a separate authority layer. Muse has Sentinel, a host-side component that sits outside the agent runtime. Muse can propose a connector action or network request, but Sentinel is the component that actually decides whether it is allowed. It evaluates connector policies and network requests independently of the agent. AWS is moving toward a similar architecture with AgentCore Gateway and AgentCore Policy. Agents request tool calls through a centralized intermediary, while authorization policies can inspect the action and even individual parameters. AWS explicitly recommends enforcing authorization outside the agent so a compromised agent cannot simply decide that it has permission. Microsoft makes essentially the same recommendation that authorization should happen on every action rather than only when the agent starts a session. High-impact actions can then require separate approval or temporary escalation. The main goal is for the model to become a proposer rather than an authority. It can decide that sending an email is the appropriate next action, but a different, more privileged component determines whether this agent, acting for this user on this task, is actually allowed to send that email to that recipient. Give the Agent Capabilities, Not Credentials If we aren't going to fully trust our agents and models, we certainly should not give them the keys to the kingdom. An agent that can see a credential can potentially leak it. The solution is to let the agent use authenticated systems without ever possessing the underlying secrets. This is not a new concept for cloud infra in general, but it's increasingly being extended to agents. Muse’s hatch-authd stores real credentials outside the agent runtime. The agent sees surrogate credentials instead. When an approved network request crosses the boundary, Sentinel obtains the actual credential and swaps it just before the request leaves the system. The model never needs to see it. NVIDIA’s Secure Agent Workspace uses almost exactly the same approach. A credential proxy rewrites authorization headers at egress. Postman’s AI Engineer similarly keeps credentials at arm's length so the agent can request authenticated actions without receiving the credentials themselves. Google applies the same idea to browser passwords. Gemini’s agentic Chrome capabilities can use Google Password Manager, but the model does not receive direct access to stored passwords. Clearly, this is stronger than telling the model not to reveal a secret because there is no secret available for it to reveal. Treat Network Egress as a Privileged Action An agent that can read private data and make arbitrary network requests has most of what it needs to leak that data and exfiltrate it. Several architectures therefore put a hard boundary around egress. The architectural idea is to treat reading data and transmitting data as separate capabilities. An agent might legitimately need access to your email, source code, CRM, or internal documents, but that does not mean every process handling that information should also be able to open an arbitrary connection to the internet. Muse routes runtime traffic through Sentinel. The authorization layer can inspect the hostname, resolved IP, port, protocol, HTTP method, path, and decoded request before allowing it to leave the VM. Agent Browser supports domain allowlists that apply not only to top-level navigation but also to subresources, WebSockets, EventSource, and sendBeacon. Muse goes even further with what Meta calls tainted egress. Processes begin clean and become tainted when they read user data. Once a process is tainted, some automatic egress privileges disappear. So it tracks whether the agent’s processes have accessed a user’s private data, and extra rules kick in. In this scenario, the request goes to a stricter approval path, perhaps requiring human approval. Meta implements this using eBPF and Linux Security Module hooks. There are likely even more creative ways we can try to observe agent behavior and restrict network actions. These types of capabilities are essential if we are going to continue giving agents more personal and confidential information. Without them, it would be all too easy to convince agents to hand over private data. Give the Model Only the Data It Needs Another approach attacks the problem earlier, at the data level. Rather than letting the model see everything and controlling what it can do afterward, the pattern is to reduce what enters its context in the first place. Google is doing this in agentic Chrome with Agent Origin Sets. For a given task, the system maintains a constrained set of web origins the agent can read or interact with. A separate gating function determines whether additional origins are relevant, and the planning model can't simply add arbitrary sites itself. Google can also distinguish between origins that are readable and origins that are writable. Suppose an agent is booking a flight. The browser may contain logged-in sessions for email, banking, healthcare, internal company tools, and dozens of other sites. A human browsing the web has access to all of them, but the agent does not necessarily need the same authority. Instead, the system can construct a task-specific view of the world. This principle extends beyond browsers. Retrieval systems can filter records before they reach the model, tools can return only authorized fields, and memory can be partitioned by user, project, or task. AWS explicitly recommends enforcing data authorization outside the agent so prompt injection cannot cause the model to retrieve information the user was never authorized to access. This seems obvious because it all comes back to just good data engineering principles, but it's just as important as the other mitigations. Controlling context can be an authorization mechanism of its own. Use Another Higher-Trust Model Not every useful security decision can be expressed as a deterministic rule. Questions like whether an action was actually implied by what the user asked, whether a navigation makes sense for the task, or whether a tool call is suspicious given the surrounding context require judgment. That has produced another architecture. If you put a second model between the main model and the action, you can use a specialized agent to mitigate some of the risk. Google’s agentic Chrome architecture includes a User Alignment Critic. The planner proposes an action, then a separate model evaluates whether that action serves the user’s stated goal. Importantly, the critic is isolated from the untrusted page content that influenced the planner. It sees metadata about the proposed action, but not the raw potentially malicious web content. Muse uses a related pattern. hatch-safety runs outside the runtime container and independently inspects inference traffic using models and classifiers for threats including prompt injection. Because those components live outside the runtime, a compromised runtime cannot simply disable them. This resembles the runtime LLM-as-judge pattern I've written about, but with a security constraint added. The judge should not necessarily have the same context as the model it is judging. If both models ingest the same malicious instructions, adding a second model may just give the attacker two models to manipulate. A higher-trust model can instead receive a smaller, sanitized representation of the proposed action and answer a narrower question. The asymmetry between the two models is what makes the architecture useful. Scope Authority to the Task Traditional applications often authenticate once and then carry substantial authority for the rest of a session. Or that authentication represents the human and not the agent. That model gets dangerous when the thing holding the session is autonomous. Microsoft recommends giving agents their own identities, restricting them to task-specific roles, using short-lived tokens, and applying just-in-time elevation rather than broad standing permissions. AWS similarly recommends temporary, scoped credentials and a traceable authorization chain between the user and every agent action. Keycard takes this idea further by giving agents their own identity and scoped permissions, so access can be granted to the agent for a particular resource or action rather than inheriting the human’s credentials and authority. Agent permissions are starting to look less like application permissions and more like leases. An agent gets enough authority to perform a specific piece of work, but then that authority expires. Selectively Put Humans at the Boundaries Human approval aka human in the loop has been the standard for a while because it is the obvious answer to model uncertainty. But requiring approval for everything eliminates much of the reason to use an agent. The more useful architecture is selective approval. Google’s browser agent can work autonomously through much of a task but asks for confirmation around sensitive sites, password use, purchases, payments, sending messages, and other consequential actions. Microsoft similarly recommends explicit approval for high-impact operations such as sends, deletes, purchases, deployments, and permission changes. The key design question is not whether the system has a human in the loop and more about where the human sits in the loop. Good checkpoints are when authority expands, sensitive information crosses a boundary, money moves, or permissions change. This is something that needs to get designed into your application stack or harness from the bottom up. Better Models Need Better Machinery Put these patterns together and a fairly coherent architecture emerges. We need a whole bunch of mostly deterministic but sometimes probabilistic machinery around our models if we are going to trust our AI-powered app. We may not fully trust the models, but that doesn't mean we can't trust what we build on top of them. The common idea is that increasingly capable models do not require us to give models increasingly broad authority. In fact, the opposite is true. As these models get more powerful, the bar is higher to design a system around them that we can trust. It's only going to get more complex from here. Attention is now focused on the systems we are building around these models and whether they can keep up with the rapid innovation in the models themselves. Follow the argument into the product model. More from The Wire /blog