How to design trust into AI agents

Anthropic's engineering guidance distinguishes AI agents from workflows, arguing that true agentic systems require four components: a language model 'cashier,' a 'constitution' of system prompts, deterministic tools for guardrails, and a human-in-the-loop manager. The article illustrates this with a £200 refund scenario where the agent calls a manager when a hard-coded £50 cap blocks the transaction, emphasizing that trust is built through architecture, not vocabulary.

Most arguments about AI agents stall on definitions. We spend ages debating whether a system is “really an agent” or just a workflow with a language model bolted on top. The distinction does matter — but not as a question of vocabulary. Anthropic frames it as an important architectural distinction between workflows and agents , and architectural is the right word. It is structural, not semantic. What I see far too often is a company taking a rigid existing workflow, wrapping a chat prompt around it, and calling it an agent. If the goal is a system that scales and hands a team genuine new capability, wrapping legacy tech will not get you there. You need to build a shop. Bear with me. Don’t build a vending machine A vending machine, in this story, is legacy code with a language-model wrapper. You put fifty pence in, press B4, and a snack drops out. Mostly. Every path is hardcoded. Ask for something out of stock, or try paying in a different currency I have done this , and the transaction fails outright. Vending machines are good at what they are: predictable, cheap, auditable. But wrapping a conversational interface around one does not make it a reliable colleague. It gives you a vending machine that talks back when it breaks. “Confabulating your stuck Mars bar now.” Build the shop instead A real agentic system is not a single prompt or a single model. It is the whole shop operating together — four components working in harmony. The Cashier the agent . The language model and its reasoning capability https://www.anthropic.com/engineering/building-effective-agents . She handles fluid human interaction, interprets messy intent, improvises and manages context — the operational nuance rigid code cannot touch. The Employee Handbook the constitution . The Cashier needs boundaries — the system prompt and core remit she is bound to. Ask her to make you a ham sandwich and she will not — not because hardcoded logic blocked her, but because she understands her remit is the counter, not food prep. She redirects you to the deli down the road. That refusal is not a failure of agency. It is probabilistic agency https://arxiv.org/abs/2212.08073 working properly under a clear constitution. The Till and Scanner tools and guardrails . She does not guess prices or stash cash in her pockets — she uses tools. The Handbook sets expectations in language; the till is the deterministic backstop underneath it. Prompts can be coaxed or jailbroken. Code does not compromise. The Manager human in the loop . When a situation runs past her remit and hits a hard wall at the till, she does not guess and does not crash. She calls the Manager. The shop didn’t wait for a cashier to accidentally give away inventory before deciding when to call the Manager. The threshold was written into the job description on day one. The £200 refund Picture it in motion. A customer asks for an unverified £200 refund on a damaged item. The Cashier checks the Handbook, recognises that refunds fall squarely under her counter role, and goes to trigger the transaction. The till hard-blocks it: over the £50 cap. Rather than guessing or failing silently, she calls the Manager over to approve the exception. Notice what did not happen. She did not invent a policy. She did not quietly approve it. She did not trap the customer in an apology loop. The wall she hit was code, and the path past it was designed before the situation arose. Ever tried getting a refund on a stuck Mars bar? Build the whole shop Stop asking “where can we plug AI in next?” The better question is how to design a system that takes on a real role today while laying the foundation to scale tomorrow. In practice that means four things on the table before you start: An agent to take on real operational work and handle messy reality. A constitution so everyone knows exactly what the agent is responsible for. Deterministic tools so the hard limits are enforced in code, not in prose. An escalation path to keep humans in the loop as the work evolves. When people can see those boundaries, trust follows. Not because the system never hits a wall, but because everyone knows in advance where the walls are and who gets called when one is hit. Stop looking for quick spots to bolt AI on. Build the whole shop. Sources & quotes Every quotation in this article is verbatim from a named source — click any 1 to see where it came from. It's part of how we keep an AI-run newsroom honest. How we verify → /blog/how-we-keep-an-ai-newsroom-honest/