The Most Important AI Agent Design Choice: Don’t Let the Model Be the Final Authority A developer argues that production AI agents should separate reasoning from execution authority, using code gates and human approval rather than letting the model be the final decision-maker. The post contrasts demo-style agents that directly chain LLM to tool with production-oriented designs that add review and approval layers, and recommends deterministic routing for known conditions. AI agents are getting very good at doing things . They can search databases, call APIs, modify tickets, draft code, update records, trigger workflows, and interact with production systems. And that changes the engineering problem. When an LLM only generates text, a bad answer is usually just that: a bad answer. When an LLM can take an action, a bad answer can become a bad state change . So the most important question in agent architecture is no longer: Can the model figure out what to do? It is: Who decides whether the model should actually be allowed to do it? Those are two very different responsibilities. And I think one of the most useful principles for production AI agents is surprisingly simple: Use the model to reason. Don’t automatically give it authority to execute. A lot of agent demos reduce to something like this: User → LLM → Tool → Action The model receives a request. It reasons about what should happen. It selects a tool. It generates the parameters. The tool executes. That is an incredibly productive abstraction. It is also a risky one when the tool can affect something real. The same probabilistic system is effectively doing two jobs: You can try to fix this with prompting: Always ask for confirmation before making important changes. But that is still an instruction. It is not a security boundary. The difference becomes clearer when you compare the two architectures. %%{init: {'theme':'base','themeVariables': { 'primaryTextColor':' 111827', 'secondaryTextColor':' 111827', 'tertiaryTextColor':' 111827', 'textColor':' 111827', 'edgeLabelBackground':' FFFFFF', 'lineColor':' 4B5563' }}}%% flowchart LR subgraph BAD "❌ Demo-Style Agent" direction LR A "User" -- B "🧠 LLM" B -- C "🔧 Tool" C -- D "💥 Real-World Action" end subgraph GOOD "✅ Production-Oriented Agent" direction LR E "User" -- F "🔎 Evidence" F -- G "🧠 LLM" G -- H "🔍 Review" H -- I "🛡️ Code Gates" I -- J "👤 Approval" J -- K "🔐 Tool" K -- L "✅ Action" end classDef bad fill: FEE2E2,stroke: DC2626,stroke-width:2px,color: 111827; classDef good fill: D1FAE5,stroke: 059669,stroke-width:2px,color: 111827; classDef ai fill: EDE9FE,stroke: 7C3AED,stroke-width:2px,color: 111827; classDef guard fill: DBEAFE,stroke: 2563EB,stroke-width:2px,color: 111827; class A,B,C,D bad; class E,F,J,K,L good; class G,H ai; class I guard; The second design has more moving parts. That is intentional. Because the system is separating: Those should not always belong to the same component. One of the easiest mistakes in AI engineering is using the model simply because the model is available. Suppose incoming tasks fall into three broad categories: Known mechanical condition ↓ Deterministic workflow Needs interpretation ↓ AI investigation High-risk or ambiguous ↓ Human review If the routing decision can be made reliably in code, make it in code. For example: python def classify task : if task.has known failure signal: return "deterministic" if task.needs investigation: return "ai investigation" return "human review" The interesting part here is the default: human review Not: let the model guess LLMs are extremely valuable when a problem genuinely requires interpretation. They do not need to become the control plane for everything around them. This has practical benefits too: Use intelligence where intelligence is actually required. If another system component needs to inspect the model's output, don't make that component parse a paragraph. Instead of asking the model to generate: I believe the likely root cause is... return something closer to: { "root cause": "...", "severity": "medium", "missing information": , "recommended actions": , "citations": } Schema-constrained output changes how the rest of the application can interact with the model. Now downstream code can make checks such as: risk ok = diagnosis.severity in {"low", "medium"} citations present = bool diagnosis.citations The model is no longer merely producing prose. It is generating typed data consumed by a larger system . That distinction becomes increasingly important as agent workflows become more complex. RAG introduces another subtle problem. Suppose an LLM cites document: issue-1842 Your application verifies: citation id in retrieved documents Great. The citation is real. But that only proves the model cited something retrieval returned. It does not prove retrieval returned something useful. Imagine the query concerns a concurrency bug, but the vector search returns three vaguely related caching incidents. All three documents are real. All three IDs are valid. The LLM can still build an extremely confident, beautifully cited, completely wrong explanation from them. So a stronger check may look more like: groundedness ok = all citation id in retrieved ids and relevance score citation id = MIN RELEVANCE SCORE for citation id in diagnosis.citations Now the system checks two different properties: Does the source exist? ↓ Provenance Is the source sufficiently relevant? ↓ Retrieval quality These are not the same thing. That leads to a broader lesson: “The model cited a real source” and “the model cited evidence that supports its claim” are different guarantees. A RAG system can be perfectly citation-valid and still be badly grounded. A common agent pattern now looks like this: LLM A ↓ Generate answer LLM B ↓ Evaluate answer "PASS" ↓ Proceed This is already better than trusting one generation blindly. But it still leaves an interesting question: Why should another probabilistic model have the final authority? A stronger architecture separates critique from enforcement . LLM A ↓ Generate proposal LLM B ↓ Critique proposal Code ↓ Apply enforceable conditions For example: groundedness ok = ... risk ok = ... permission ok = ... approved = groundedness ok and risk ok and permission ok The reviewer model can still produce something very valuable: This diagnosis appears weak because the cited evidence does not fully support the proposed root cause... That explanation is useful to a human. But the system does not need to parse: APPROVE from the model's response and treat that string as authority. The distinction is simple: Let the model explain. Let deterministic systems enforce. This becomes especially important for conditions like: These are usually better represented as explicit program state than as natural-language judgment. Imagine your workflow graph contains: review → approval → execute Everything looks safe. But six months later someone refactors the graph. A shortcut gets introduced: review → execute If approval existed only as orchestration logic, you just removed the safety control by changing one edge. A stronger design puts the check inside the function that performs the mutation. python def execute state : if not state.get "approved" : raise PermissionError "Execution requires explicit approval." perform action Now you have two protections. The graph says: You should not reach execute yet. The execution boundary says: Even if you reach me, I refuse to run. That is defense in depth. And this idea generalizes far beyond AI. Security-sensitive properties should ideally be enforced as close as possible to the resource being protected. Many systems technically have a human approval screen. But underneath, the implementation is surprisingly fragile. Maybe the workflow state exists only in memory. Maybe the process is just waiting. Maybe the exact action gets regenerated after approval. A stronger human-in-the-loop design looks like this: Agent proposes action ↓ Workflow suspends ↓ minutes / hours / days ↓ Human approves ↓ The exact approved action executes This creates an infrastructure requirement that is easy to miss: the state of the paused workflow must survive independently of the application process. If the application container disappears, the approval state must not disappear with it. Conceptually: Agent Runtime ↓ Checkpoint ↓ Persistent Storage That lets the process restart completely while the workflow remains resumable. This matters in real deployments because: A human approval system that only works while one Python process stays alive is not really durable human approval. There is another subtle detail here. Suppose the agent presents this to the user: I propose posting comment X. The human approves it. Then the application asks the LLM: Generate the final comment. That creates a new output. The human never approved the new output. Instead, approval should usually bind to a concrete proposed action: proposed action = build action state approved = wait for human proposed action if approved: execute proposed action No regeneration. No reinterpretation. No second chance for model variance. The artifact the human reviews should be the artifact that crosses the mutation boundary. Once you combine these ideas, the architecture starts looking less like a chatbot with tools and more like a proper software system. %%{init: {'theme':'base','themeVariables': { 'primaryTextColor':' 111827', 'secondaryTextColor':' 111827', 'tertiaryTextColor':' 111827', 'textColor':' 111827', 'edgeLabelBackground':' FFFFFF', 'lineColor':' 4B5563' }}}%% flowchart TD A "📥 User Request / Event" -- B "🔎 Gather Evidence" B -- C{"🧭 Deterministic Classification"} C -- |"Known / Mechanical"| D "⚙️ Deterministic Path" C -- |"Needs Investigation"| E "🧠 LLM Reasoning" C -- |"Ambiguous / High Risk"| H "👤 Human Review" E -- F "🔍 Independent LLM Review" F -- G{"🛡️ Code-Enforced Gates"} G -- |"Grounded ✓