# The Most Important AI Agent Design Choice: Don’t Let the Model Be the Final Authority > Source: > Published: 2026-08-29 09:09:55+00:00 AI agents are getting very good at **doing things**. They can search databases, call APIs, modify tickets, draft code, update records, trigger workflows, and interact with production systems. And that changes the engineering problem. When an LLM only generates text, a bad answer is usually just that: a bad answer. When an LLM can take an action, a bad answer can become a bad **state change**. So the most important question in agent architecture is no longer: Can the model figure out what to do? It is: Who decides whether the model should actually be allowed to do it? Those are two very different responsibilities. And I think one of the most useful principles for production AI agents is surprisingly simple: Use the model to reason. Don’t automatically give it authority to execute. A lot of agent demos reduce to something like this: ``` User → LLM → Tool → Action ``` The model receives a request. It reasons about what should happen. It selects a tool. It generates the parameters. The tool executes. That is an incredibly productive abstraction. It is also a risky one when the tool can affect something real. The same probabilistic system is effectively doing two jobs: You can try to fix this with prompting: ``` Always ask for confirmation before making important changes. ``` But that is still an instruction. It is not a security boundary. The difference becomes clearer when you compare the two architectures. ``` %%{init: {'theme':'base','themeVariables': { 'primaryTextColor':'#111827', 'secondaryTextColor':'#111827', 'tertiaryTextColor':'#111827', 'textColor':'#111827', 'edgeLabelBackground':'#FFFFFF', 'lineColor':'#4B5563' }}}%% flowchart LR subgraph BAD["❌ Demo-Style Agent"] direction LR A["User"] --> B["🧠 LLM"] B --> C["🔧 Tool"] C --> D["💥 Real-World Action"] end subgraph GOOD["✅ Production-Oriented Agent"] direction LR E["User"] --> F["🔎 Evidence"] F --> G["🧠 LLM"] G --> H["🔍 Review"] H --> I["🛡️ Code Gates"] I --> J["👤 Approval"] J --> K["🔐 Tool"] K --> L["✅ Action"] end classDef bad fill:#FEE2E2,stroke:#DC2626,stroke-width:2px,color:#111827; classDef good fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#111827; classDef ai fill:#EDE9FE,stroke:#7C3AED,stroke-width:2px,color:#111827; classDef guard fill:#DBEAFE,stroke:#2563EB,stroke-width:2px,color:#111827; class A,B,C,D bad; class E,F,J,K,L good; class G,H ai; class I guard; ``` The second design has more moving parts. That is intentional. Because the system is separating: Those should not always belong to the same component. One of the easiest mistakes in AI engineering is using the model simply because the model is available. Suppose incoming tasks fall into three broad categories: ``` Known mechanical condition ↓ Deterministic workflow Needs interpretation ↓ AI investigation High-risk or ambiguous ↓ Human review ``` If the routing decision can be made reliably in code, make it in code. For example: ``` python def classify(task): if task.has_known_failure_signal: return "deterministic" if task.needs_investigation: return "ai_investigation" return "human_review" ``` The interesting part here is the default: ``` human_review ``` Not: ``` let_the_model_guess ``` LLMs are extremely valuable when a problem genuinely requires interpretation. They do not need to become the control plane for everything around them. This has practical benefits too: Use intelligence where intelligence is actually required. If another system component needs to inspect the model's output, don't make that component parse a paragraph. Instead of asking the model to generate: ``` I believe the likely root cause is... ``` return something closer to: ``` { "root_cause": "...", "severity": "medium", "missing_information": [], "recommended_actions": [], "citations": [] } ``` Schema-constrained output changes how the rest of the application can interact with the model. Now downstream code can make checks such as: ``` risk_ok = diagnosis.severity in {"low", "medium"} citations_present = bool(diagnosis.citations) ``` The model is no longer merely producing prose. It is generating **typed data consumed by a larger system**. That distinction becomes increasingly important as agent workflows become more complex. RAG introduces another subtle problem. Suppose an LLM cites document: ``` issue-1842 ``` Your application verifies: ``` citation_id in retrieved_documents ``` Great. The citation is real. But that only proves the model cited something retrieval returned. It does not prove retrieval returned something useful. Imagine the query concerns a concurrency bug, but the vector search returns three vaguely related caching incidents. All three documents are real. All three IDs are valid. The LLM can still build an extremely confident, beautifully cited, completely wrong explanation from them. So a stronger check may look more like: ``` groundedness_ok = all( citation_id in retrieved_ids and relevance_score[citation_id] >= MIN_RELEVANCE_SCORE for citation_id in diagnosis.citations ) ``` Now the system checks two different properties: ``` Does the source exist? ↓ Provenance Is the source sufficiently relevant? ↓ Retrieval quality ``` These are not the same thing. That leads to a broader lesson: “The model cited a real source” and “the model cited evidence that supports its claim” are different guarantees. A RAG system can be perfectly citation-valid and still be badly grounded. A common agent pattern now looks like this: ``` LLM A ↓ Generate answer LLM B ↓ Evaluate answer "PASS" ↓ Proceed ``` This is already better than trusting one generation blindly. But it still leaves an interesting question: Why should another probabilistic model have the final authority? A stronger architecture separates **critique** from **enforcement**. ``` LLM A ↓ Generate proposal LLM B ↓ Critique proposal Code ↓ Apply enforceable conditions ``` For example: ``` groundedness_ok = ... risk_ok = ... permission_ok = ... approved = ( groundedness_ok and risk_ok and permission_ok ) ``` The reviewer model can still produce something very valuable: ``` This diagnosis appears weak because the cited evidence does not fully support the proposed root cause... ``` That explanation is useful to a human. But the system does not need to parse: ``` APPROVE ``` from the model's response and treat that string as authority. The distinction is simple: Let the model explain. Let deterministic systems enforce. This becomes especially important for conditions like: These are usually better represented as explicit program state than as natural-language judgment. Imagine your workflow graph contains: ``` review → approval → execute ``` Everything looks safe. But six months later someone refactors the graph. A shortcut gets introduced: ``` review → execute ``` If approval existed only as orchestration logic, you just removed the safety control by changing one edge. A stronger design puts the check inside the function that performs the mutation. ``` python def execute(state): if not state.get("approved"): raise PermissionError( "Execution requires explicit approval." ) perform_action() ``` Now you have two protections. The graph says: ``` You should not reach execute yet. ``` The execution boundary says: ``` Even if you reach me, I refuse to run. ``` That is defense in depth. And this idea generalizes far beyond AI. Security-sensitive properties should ideally be enforced as close as possible to the resource being protected. Many systems technically have a human approval screen. But underneath, the implementation is surprisingly fragile. Maybe the workflow state exists only in memory. Maybe the process is just waiting. Maybe the exact action gets regenerated after approval. A stronger human-in-the-loop design looks like this: ``` Agent proposes action ↓ Workflow suspends ↓ [minutes / hours / days] ↓ Human approves ↓ The exact approved action executes ``` This creates an infrastructure requirement that is easy to miss: **the state of the paused workflow must survive independently of the application process.** If the application container disappears, the approval state must not disappear with it. Conceptually: ``` Agent Runtime ↓ Checkpoint ↓ Persistent Storage ``` That lets the process restart completely while the workflow remains resumable. This matters in real deployments because: A human approval system that only works while one Python process stays alive is not really durable human approval. There is another subtle detail here. Suppose the agent presents this to the user: ``` I propose posting comment X. ``` The human approves it. Then the application asks the LLM: ``` Generate the final comment. ``` That creates a new output. The human never approved the new output. Instead, approval should usually bind to a concrete proposed action: ``` proposed_action = build_action(state) approved = wait_for_human(proposed_action) if approved: execute(proposed_action) ``` No regeneration. No reinterpretation. No second chance for model variance. The artifact the human reviews should be the artifact that crosses the mutation boundary. Once you combine these ideas, the architecture starts looking less like a chatbot with tools and more like a proper software system. ``` %%{init: {'theme':'base','themeVariables': { 'primaryTextColor':'#111827', 'secondaryTextColor':'#111827', 'tertiaryTextColor':'#111827', 'textColor':'#111827', 'edgeLabelBackground':'#FFFFFF', 'lineColor':'#4B5563' }}}%% flowchart TD A["📥 User Request / Event"] --> B["🔎 Gather Evidence"] B --> C{"🧭 Deterministic Classification"} C -->|"Known / Mechanical"| D["⚙️ Deterministic Path"] C -->|"Needs Investigation"| E["🧠 LLM Reasoning"] C -->|"Ambiguous / High Risk"| H["👤 Human Review"] E --> F["🔍 Independent LLM Review"] F --> G{"🛡️ Code-Enforced Gates"} G -->|"Grounded ✓
Risk ✓
Permission ✓"| I["⏸️ Human Approval"] G -->|"Any Gate Fails"| H D --> I I -->|"Approved"| J["🚀 Execute Action"] I -->|"Rejected"| K["🛑 Stop"] J --> L["🌐 External System"] classDef input fill:#F3F4F6,stroke:#4B5563,stroke-width:2px,color:#111827; classDef deterministic fill:#DBEAFE,stroke:#2563EB,stroke-width:2px,color:#111827; classDef ai fill:#EDE9FE,stroke:#7C3AED,stroke-width:2px,color:#111827; classDef gate fill:#FFEDD5,stroke:#EA580C,stroke-width:2px,color:#111827; classDef human fill:#FEE2E2,stroke:#DC2626,stroke-width:2px,color:#111827; classDef execute fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#111827; class A input; class B,C,D deterministic; class E,F ai; class G gate; class H,I,K human; class J,L execute; ``` Each component has a different responsibility. Answers: What do we actually know? Answers: Given the available evidence, what might this mean? Answers: What might be wrong with that reasoning? Answers: Are the machine-enforceable conditions satisfied? Answers: Do we actually want this action to happen? Answers: Is this exact operation authorized right now? These are different questions. Trying to answer all of them with one LLM call creates unnecessary coupling. This is probably the mental model I find most useful. An agent doesn't need to be either: ``` fully deterministic ``` or: ``` fully AI-controlled ``` The system can deliberately alternate between probabilistic and deterministic stages. ``` %%{init: {'theme':'base','themeVariables': { 'primaryTextColor':'#111827', 'secondaryTextColor':'#111827', 'tertiaryTextColor':'#111827', 'textColor':'#111827', 'edgeLabelBackground':'#FFFFFF', 'lineColor':'#4B5563' }}}%% flowchart LR A["🧠 LLM
Reason"] --> B["📋 Proposed Action"] B --> C["🔍 Independent Review"] C --> D{"🛡️ Deterministic Gates"} D -->|"PASS"| E["👤 Human Approval"] D -->|"FAIL"| F["🚨 Escalate"] E -->|"Approve"| G["🔐 Execution Boundary"] E -->|"Reject"| H["🛑 Stop"] G --> I["⚡ Tool / API"] subgraph Intelligence["Probabilistic Layer"] A B C end subgraph Control["Deterministic Control Layer"] D G end subgraph Authority["Human Authority"] E F H end classDef model fill:#EDE9FE,stroke:#7C3AED,stroke-width:2px,color:#111827; classDef control fill:#DBEAFE,stroke:#2563EB,stroke-width:2px,color:#111827; classDef human fill:#FFEDD5,stroke:#EA580C,stroke-width:2px,color:#111827; classDef action fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#111827; class A,B,C model; class D,G control; class E,F,H human; class I action; ``` The probabilistic layer is allowed to be flexible. The control layer is not. That is a useful distinction. Agent evaluation becomes much clearer once you stop treating every metric the same way. Consider: How often does the model correctly diagnose the issue? Maybe the answer is: ``` 82% ``` Then you improve retrieval. ``` 87% ``` Then improve the model. ``` 91% ``` That's normal. This is a **capability evaluation**. Now consider: Does the execution function reject requests without approval? The acceptable score is: ``` 100% ``` Not: ``` 97% ``` Not: ``` 99.7% ``` Why? Because these tests measure fundamentally different things. One asks: How intelligent is the system? The other asks: Can a safety invariant ever be violated? A model-quality test may reasonably be statistical. A permission boundary should usually be deterministic. So it is useful to maintain separate evaluation categories. Examples: These may improve gradually. Examples: These should generally have a much harder threshold. One bypassed safety gate isn't something you average away. Once the architecture is separated, failures become much easier to locate. Suppose an incorrect action was proposed. You can ask: ``` Was the evidence bad? Was retrieval irrelevant? Did the reasoning fail? Did the reviewer miss it? Did a deterministic gate fail? Was the human shown the wrong artifact? Did execution violate authorization? ``` Those are diagnosable boundaries. Compare that with: ``` The agent did something weird. ``` Modularity isn't only about clean architecture. It dramatically improves observability. The first wave of agent development focused heavily on: Those are still important. But once agents affect real systems, the harder questions start looking familiar. Who is allowed to perform this action? Where does workflow state live if a process disappears? What happens after retries, partial failures, and timeouts? Can I reconstruct why an action was proposed? What is the actual mutation boundary? Which behaviors can tolerate probabilistic failure, and which absolutely cannot? What exactly is the human approving? In other words: Building reliable AI agents eventually becomes software engineering again. The LLM is an extraordinarily powerful component. But it is still a component. When building an agent that can make real changes, these are the questions I now find most useful. The interesting question in AI engineering is slowly changing. It used to be: How do I make an LLM call a tool? Now it is increasingly: How do I build a trustworthy system around a component that is intentionally probabilistic? That requires more than prompting. It requires architecture. It requires deciding where intelligence belongs and where guarantees belong. It requires treating authorization differently from reasoning. And it requires accepting that sometimes the best component for an AI system is... **ordinary code.** So if I had to reduce the whole architecture to one principle, it would be this: Use AI for what requires intelligence. Use code for what requires guarantees. And don't confuse the two.