{"slug": "the-most-important-ai-agent-design-choice-dont-let-the-model-be-the-final", "title": "The Most Important AI Agent Design Choice: Don’t Let the Model Be the Final Authority", "summary": "A developer argues that production AI agents should separate reasoning from execution authority, using code gates and human approval rather than letting the model be the final decision-maker. The post contrasts demo-style agents that directly chain LLM to tool with production-oriented designs that add review and approval layers, and recommends deterministic routing for known conditions.", "body_md": "AI agents are getting very good at **doing things**.\n\nThey can search databases, call APIs, modify tickets, draft code, update records, trigger workflows, and interact with production systems.\n\nAnd that changes the engineering problem.\n\nWhen an LLM only generates text, a bad answer is usually just that: a bad answer.\n\nWhen an LLM can take an action, a bad answer can become a bad **state change**.\n\nSo the most important question in agent architecture is no longer:\n\nCan the model figure out what to do?\n\nIt is:\n\nWho decides whether the model should actually be allowed to do it?\n\nThose are two very different responsibilities.\n\nAnd I think one of the most useful principles for production AI agents is surprisingly simple:\n\nUse the model to reason. Don’t automatically give it authority to execute.\n\nA lot of agent demos reduce to something like this:\n\n```\nUser → LLM → Tool → Action\n```\n\nThe model receives a request.\n\nIt reasons about what should happen.\n\nIt selects a tool.\n\nIt generates the parameters.\n\nThe tool executes.\n\nThat is an incredibly productive abstraction.\n\nIt is also a risky one when the tool can affect something real.\n\nThe same probabilistic system is effectively doing two jobs:\n\nYou can try to fix this with prompting:\n\n```\nAlways ask for confirmation before making important changes.\n```\n\nBut that is still an instruction.\n\nIt is not a security boundary.\n\nThe difference becomes clearer when you compare the two architectures.\n\n```\n%%{init: {'theme':'base','themeVariables': {\n'primaryTextColor':'#111827',\n'secondaryTextColor':'#111827',\n'tertiaryTextColor':'#111827',\n'textColor':'#111827',\n'edgeLabelBackground':'#FFFFFF',\n'lineColor':'#4B5563'\n}}}%%\n\nflowchart LR\n\n    subgraph BAD[\"❌ Demo-Style Agent\"]\n        direction LR\n        A[\"User\"] --> B[\"🧠 LLM\"]\n        B --> C[\"🔧 Tool\"]\n        C --> D[\"💥 Real-World Action\"]\n    end\n\n    subgraph GOOD[\"✅ Production-Oriented Agent\"]\n        direction LR\n        E[\"User\"] --> F[\"🔎 Evidence\"]\n        F --> G[\"🧠 LLM\"]\n        G --> H[\"🔍 Review\"]\n        H --> I[\"🛡️ Code Gates\"]\n        I --> J[\"👤 Approval\"]\n        J --> K[\"🔐 Tool\"]\n        K --> L[\"✅ Action\"]\n    end\n\n    classDef bad fill:#FEE2E2,stroke:#DC2626,stroke-width:2px,color:#111827;\n    classDef good fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#111827;\n    classDef ai fill:#EDE9FE,stroke:#7C3AED,stroke-width:2px,color:#111827;\n    classDef guard fill:#DBEAFE,stroke:#2563EB,stroke-width:2px,color:#111827;\n\n    class A,B,C,D bad;\n    class E,F,J,K,L good;\n    class G,H ai;\n    class I guard;\n```\n\nThe second design has more moving parts.\n\nThat is intentional.\n\nBecause the system is separating:\n\nThose should not always belong to the same component.\n\nOne of the easiest mistakes in AI engineering is using the model simply because the model is available.\n\nSuppose incoming tasks fall into three broad categories:\n\n```\nKnown mechanical condition\n        ↓\nDeterministic workflow\n\nNeeds interpretation\n        ↓\nAI investigation\n\nHigh-risk or ambiguous\n        ↓\nHuman review\n```\n\nIf the routing decision can be made reliably in code, make it in code.\n\nFor example:\n\n``` python\ndef classify(task):\n    if task.has_known_failure_signal:\n        return \"deterministic\"\n\n    if task.needs_investigation:\n        return \"ai_investigation\"\n\n    return \"human_review\"\n```\n\nThe interesting part here is the default:\n\n```\nhuman_review\n```\n\nNot:\n\n```\nlet_the_model_guess\n```\n\nLLMs are extremely valuable when a problem genuinely requires interpretation.\n\nThey do not need to become the control plane for everything around them.\n\nThis has practical benefits too:\n\nUse intelligence where intelligence is actually required.\n\nIf another system component needs to inspect the model's output, don't make that component parse a paragraph.\n\nInstead of asking the model to generate:\n\n```\nI believe the likely root cause is...\n```\n\nreturn something closer to:\n\n```\n{\n  \"root_cause\": \"...\",\n  \"severity\": \"medium\",\n  \"missing_information\": [],\n  \"recommended_actions\": [],\n  \"citations\": []\n}\n```\n\nSchema-constrained output changes how the rest of the application can interact with the model.\n\nNow downstream code can make checks such as:\n\n```\nrisk_ok = diagnosis.severity in {\"low\", \"medium\"}\n\ncitations_present = bool(diagnosis.citations)\n```\n\nThe model is no longer merely producing prose.\n\nIt is generating **typed data consumed by a larger system**.\n\nThat distinction becomes increasingly important as agent workflows become more complex.\n\nRAG introduces another subtle problem.\n\nSuppose an LLM cites document:\n\n```\nissue-1842\n```\n\nYour application verifies:\n\n```\ncitation_id in retrieved_documents\n```\n\nGreat.\n\nThe citation is real.\n\nBut that only proves the model cited something retrieval returned.\n\nIt does not prove retrieval returned something useful.\n\nImagine the query concerns a concurrency bug, but the vector search returns three vaguely related caching incidents.\n\nAll three documents are real.\n\nAll three IDs are valid.\n\nThe LLM can still build an extremely confident, beautifully cited, completely wrong explanation from them.\n\nSo a stronger check may look more like:\n\n```\ngroundedness_ok = all(\n    citation_id in retrieved_ids\n    and relevance_score[citation_id] >= MIN_RELEVANCE_SCORE\n    for citation_id in diagnosis.citations\n)\n```\n\nNow the system checks two different properties:\n\n```\nDoes the source exist?\n        ↓\nProvenance\n\nIs the source sufficiently relevant?\n        ↓\nRetrieval quality\n```\n\nThese are not the same thing.\n\nThat leads to a broader lesson:\n\n“The model cited a real source” and “the model cited evidence that supports its claim” are different guarantees.\n\nA RAG system can be perfectly citation-valid and still be badly grounded.\n\nA common agent pattern now looks like this:\n\n```\nLLM A\n  ↓\nGenerate answer\n\nLLM B\n  ↓\nEvaluate answer\n\n\"PASS\"\n  ↓\nProceed\n```\n\nThis is already better than trusting one generation blindly.\n\nBut it still leaves an interesting question:\n\nWhy should another probabilistic model have the final authority?\n\nA stronger architecture separates **critique** from **enforcement**.\n\n```\nLLM A\n  ↓\nGenerate proposal\n\nLLM B\n  ↓\nCritique proposal\n\nCode\n  ↓\nApply enforceable conditions\n```\n\nFor example:\n\n```\ngroundedness_ok = ...\nrisk_ok = ...\npermission_ok = ...\n\napproved = (\n    groundedness_ok\n    and risk_ok\n    and permission_ok\n)\n```\n\nThe reviewer model can still produce something very valuable:\n\n```\nThis diagnosis appears weak because the cited evidence does\nnot fully support the proposed root cause...\n```\n\nThat explanation is useful to a human.\n\nBut the system does not need to parse:\n\n```\nAPPROVE\n```\n\nfrom the model's response and treat that string as authority.\n\nThe distinction is simple:\n\nLet the model explain. Let deterministic systems enforce.\n\nThis becomes especially important for conditions like:\n\nThese are usually better represented as explicit program state than as natural-language judgment.\n\nImagine your workflow graph contains:\n\n```\nreview → approval → execute\n```\n\nEverything looks safe.\n\nBut six months later someone refactors the graph.\n\nA shortcut gets introduced:\n\n```\nreview → execute\n```\n\nIf approval existed only as orchestration logic, you just removed the safety control by changing one edge.\n\nA stronger design puts the check inside the function that performs the mutation.\n\n``` python\ndef execute(state):\n    if not state.get(\"approved\"):\n        raise PermissionError(\n            \"Execution requires explicit approval.\"\n        )\n\n    perform_action()\n```\n\nNow you have two protections.\n\nThe graph says:\n\n```\nYou should not reach execute yet.\n```\n\nThe execution boundary says:\n\n```\nEven if you reach me, I refuse to run.\n```\n\nThat is defense in depth.\n\nAnd this idea generalizes far beyond AI.\n\nSecurity-sensitive properties should ideally be enforced as close as possible to the resource being protected.\n\nMany systems technically have a human approval screen.\n\nBut underneath, the implementation is surprisingly fragile.\n\nMaybe the workflow state exists only in memory.\n\nMaybe the process is just waiting.\n\nMaybe the exact action gets regenerated after approval.\n\nA stronger human-in-the-loop design looks like this:\n\n```\nAgent proposes action\n        ↓\n\nWorkflow suspends\n        ↓\n\n[minutes / hours / days]\n\n        ↓\nHuman approves\n        ↓\n\nThe exact approved action executes\n```\n\nThis creates an infrastructure requirement that is easy to miss:\n\n**the state of the paused workflow must survive independently of the application process.**\n\nIf the application container disappears, the approval state must not disappear with it.\n\nConceptually:\n\n```\nAgent Runtime\n     ↓\nCheckpoint\n     ↓\nPersistent Storage\n```\n\nThat lets the process restart completely while the workflow remains resumable.\n\nThis matters in real deployments because:\n\nA human approval system that only works while one Python process stays alive is not really durable human approval.\n\nThere is another subtle detail here.\n\nSuppose the agent presents this to the user:\n\n```\nI propose posting comment X.\n```\n\nThe human approves it.\n\nThen the application asks the LLM:\n\n```\nGenerate the final comment.\n```\n\nThat creates a new output.\n\nThe human never approved the new output.\n\nInstead, approval should usually bind to a concrete proposed action:\n\n```\nproposed_action = build_action(state)\n\napproved = wait_for_human(proposed_action)\n\nif approved:\n    execute(proposed_action)\n```\n\nNo regeneration.\n\nNo reinterpretation.\n\nNo second chance for model variance.\n\nThe artifact the human reviews should be the artifact that crosses the mutation boundary.\n\nOnce you combine these ideas, the architecture starts looking less like a chatbot with tools and more like a proper software system.\n\n```\n%%{init: {'theme':'base','themeVariables': {\n'primaryTextColor':'#111827',\n'secondaryTextColor':'#111827',\n'tertiaryTextColor':'#111827',\n'textColor':'#111827',\n'edgeLabelBackground':'#FFFFFF',\n'lineColor':'#4B5563'\n}}}%%\n\nflowchart TD\n    A[\"📥 User Request / Event\"] --> B[\"🔎 Gather Evidence\"]\n    B --> C{\"🧭 Deterministic Classification\"}\n\n    C -->|\"Known / Mechanical\"| D[\"⚙️ Deterministic Path\"]\n    C -->|\"Needs Investigation\"| E[\"🧠 LLM Reasoning\"]\n    C -->|\"Ambiguous / High Risk\"| H[\"👤 Human Review\"]\n\n    E --> F[\"🔍 Independent LLM Review\"]\n    F --> G{\"🛡️ Code-Enforced Gates\"}\n\n    G -->|\"Grounded ✓<br/>Risk ✓<br/>Permission ✓\"| I[\"⏸️ Human Approval\"]\n    G -->|\"Any Gate Fails\"| H\n\n    D --> I\n\n    I -->|\"Approved\"| J[\"🚀 Execute Action\"]\n    I -->|\"Rejected\"| K[\"🛑 Stop\"]\n\n    J --> L[\"🌐 External System\"]\n\n    classDef input fill:#F3F4F6,stroke:#4B5563,stroke-width:2px,color:#111827;\n    classDef deterministic fill:#DBEAFE,stroke:#2563EB,stroke-width:2px,color:#111827;\n    classDef ai fill:#EDE9FE,stroke:#7C3AED,stroke-width:2px,color:#111827;\n    classDef gate fill:#FFEDD5,stroke:#EA580C,stroke-width:2px,color:#111827;\n    classDef human fill:#FEE2E2,stroke:#DC2626,stroke-width:2px,color:#111827;\n    classDef execute fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#111827;\n\n    class A input;\n    class B,C,D deterministic;\n    class E,F ai;\n    class G gate;\n    class H,I,K human;\n    class J,L execute;\n```\n\nEach component has a different responsibility.\n\nAnswers:\n\nWhat do we actually know?\n\nAnswers:\n\nGiven the available evidence, what might this mean?\n\nAnswers:\n\nWhat might be wrong with that reasoning?\n\nAnswers:\n\nAre the machine-enforceable conditions satisfied?\n\nAnswers:\n\nDo we actually want this action to happen?\n\nAnswers:\n\nIs this exact operation authorized right now?\n\nThese are different questions.\n\nTrying to answer all of them with one LLM call creates unnecessary coupling.\n\nThis is probably the mental model I find most useful.\n\nAn agent doesn't need to be either:\n\n```\nfully deterministic\n```\n\nor:\n\n```\nfully AI-controlled\n```\n\nThe system can deliberately alternate between probabilistic and deterministic stages.\n\n```\n%%{init: {'theme':'base','themeVariables': {\n'primaryTextColor':'#111827',\n'secondaryTextColor':'#111827',\n'tertiaryTextColor':'#111827',\n'textColor':'#111827',\n'edgeLabelBackground':'#FFFFFF',\n'lineColor':'#4B5563'\n}}}%%\n\nflowchart LR\n    A[\"🧠 LLM<br/>Reason\"] --> B[\"📋 Proposed Action\"]\n\n    B --> C[\"🔍 Independent Review\"]\n    C --> D{\"🛡️ Deterministic Gates\"}\n\n    D -->|\"PASS\"| E[\"👤 Human Approval\"]\n    D -->|\"FAIL\"| F[\"🚨 Escalate\"]\n\n    E -->|\"Approve\"| G[\"🔐 Execution Boundary\"]\n    E -->|\"Reject\"| H[\"🛑 Stop\"]\n\n    G --> I[\"⚡ Tool / API\"]\n\n    subgraph Intelligence[\"Probabilistic Layer\"]\n        A\n        B\n        C\n    end\n\n    subgraph Control[\"Deterministic Control Layer\"]\n        D\n        G\n    end\n\n    subgraph Authority[\"Human Authority\"]\n        E\n        F\n        H\n    end\n\n    classDef model fill:#EDE9FE,stroke:#7C3AED,stroke-width:2px,color:#111827;\n    classDef control fill:#DBEAFE,stroke:#2563EB,stroke-width:2px,color:#111827;\n    classDef human fill:#FFEDD5,stroke:#EA580C,stroke-width:2px,color:#111827;\n    classDef action fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#111827;\n\n    class A,B,C model;\n    class D,G control;\n    class E,F,H human;\n    class I action;\n```\n\nThe probabilistic layer is allowed to be flexible.\n\nThe control layer is not.\n\nThat is a useful distinction.\n\nAgent evaluation becomes much clearer once you stop treating every metric the same way.\n\nConsider:\n\nHow often does the model correctly diagnose the issue?\n\nMaybe the answer is:\n\n```\n82%\n```\n\nThen you improve retrieval.\n\n```\n87%\n```\n\nThen improve the model.\n\n```\n91%\n```\n\nThat's normal.\n\nThis is a **capability evaluation**.\n\nNow consider:\n\nDoes the execution function reject requests without approval?\n\nThe acceptable score is:\n\n```\n100%\n```\n\nNot:\n\n```\n97%\n```\n\nNot:\n\n```\n99.7%\n```\n\nWhy?\n\nBecause these tests measure fundamentally different things.\n\nOne asks:\n\nHow intelligent is the system?\n\nThe other asks:\n\nCan a safety invariant ever be violated?\n\nA model-quality test may reasonably be statistical.\n\nA permission boundary should usually be deterministic.\n\nSo it is useful to maintain separate evaluation categories.\n\nExamples:\n\nThese may improve gradually.\n\nExamples:\n\nThese should generally have a much harder threshold.\n\nOne bypassed safety gate isn't something you average away.\n\nOnce the architecture is separated, failures become much easier to locate.\n\nSuppose an incorrect action was proposed.\n\nYou can ask:\n\n```\nWas the evidence bad?\n\nWas retrieval irrelevant?\n\nDid the reasoning fail?\n\nDid the reviewer miss it?\n\nDid a deterministic gate fail?\n\nWas the human shown the wrong artifact?\n\nDid execution violate authorization?\n```\n\nThose are diagnosable boundaries.\n\nCompare that with:\n\n```\nThe agent did something weird.\n```\n\nModularity isn't only about clean architecture.\n\nIt dramatically improves observability.\n\nThe first wave of agent development focused heavily on:\n\nThose are still important.\n\nBut once agents affect real systems, the harder questions start looking familiar.\n\nWho is allowed to perform this action?\n\nWhere does workflow state live if a process disappears?\n\nWhat happens after retries, partial failures, and timeouts?\n\nCan I reconstruct why an action was proposed?\n\nWhat is the actual mutation boundary?\n\nWhich behaviors can tolerate probabilistic failure, and which absolutely cannot?\n\nWhat exactly is the human approving?\n\nIn other words:\n\nBuilding reliable AI agents eventually becomes software engineering again.\n\nThe LLM is an extraordinarily powerful component.\n\nBut it is still a component.\n\nWhen building an agent that can make real changes, these are the questions I now find most useful.\n\nThe interesting question in AI engineering is slowly changing.\n\nIt used to be:\n\nHow do I make an LLM call a tool?\n\nNow it is increasingly:\n\nHow do I build a trustworthy system around a component that is intentionally probabilistic?\n\nThat requires more than prompting.\n\nIt requires architecture.\n\nIt requires deciding where intelligence belongs and where guarantees belong.\n\nIt requires treating authorization differently from reasoning.\n\nAnd it requires accepting that sometimes the best component for an AI system is...\n\n**ordinary code.**\n\nSo if I had to reduce the whole architecture to one principle, it would be this:\n\nUse AI for what requires intelligence. Use code for what requires guarantees.\n\nAnd don't confuse the two.", "url": "https://wpnews.pro/news/the-most-important-ai-agent-design-choice-dont-let-the-model-be-the-final", "canonical_source": "https://dev.to/officialbidisha/the-most-important-ai-agent-design-choice-dont-let-the-model-be-the-final-authority-1lj0", "published_at": "2026-08-29 09:09:55+00:00", "updated_at": "2026-08-29 09:18:54.581332+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-infrastructure", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/the-most-important-ai-agent-design-choice-dont-let-the-model-be-the-final", "markdown": "https://wpnews.pro/news/the-most-important-ai-agent-design-choice-dont-let-the-model-be-the-final.md", "text": "https://wpnews.pro/news/the-most-important-ai-agent-design-choice-dont-let-the-model-be-the-final.txt", "jsonld": "https://wpnews.pro/news/the-most-important-ai-agent-design-choice-dont-let-the-model-be-the-final.jsonld"}}