Human Control Cannot Be a Checkbox in an Autonomous Agent System

A developer argues that human approval buttons in autonomous agent systems are often ceremonial rather than true control boundaries, and outlines five properties required for operational human control, including role differentiation, pre-execution authorization, and evidence-based review. The piece connects these requirements to NIST's AI Risk Management Framework and the EU AI Act.

An AI agent prepares a production deployment. It has changed twelve files, run several checks, and produced a confident summary. The interface now shows a green button: Human approval required The operator clicks Approve . It looks like a human remained in control. But what did that click actually mean? Did the operator see the exact patch? Were all required checks executed? Did the agent disclose what it could not measure? Was deployment still blocked before the click? Could the operator reject or narrow the action? Can anyone later prove which person approved which version under which authority? If those questions have no reliable answer, the button is not a control boundary. It is a ceremony. As AI agents move from generating text to changing code, sending messages, purchasing services, and operating business systems, this distinction becomes important: A button is an interface. Authority is a property of the system. Many agent products add a human approval step near a consequential action. This is often useful, but the design can become superficial. The weak pattern looks like this: The person is present, but presence alone does not establish control. The approval may be bundled across unrelated actions. The summary may omit uncertainty. The person may not be the party authorized to accept the risk. The action may already have started. The audit log may be editable by the same system that produced it. The approval may not be bound to the exact request that was executed. This produces a dangerous ambiguity: the organization can say a human approved the action, while the human cannot prove what they were asked to approve. Human control becomes operational only when the system preserves five properties. "A human" is not a role. The system should identify who has the right to authorize a capability, who may execute work, who verifies evidence, and who accepts the result. These can be different people or systems. This matters because the developer who built an agent should not automatically become the customer who accepts its delivery. Likewise, an agent should not be allowed to execute, verify, and accept its own work simply because all three steps can be automated. NIST's AI Risk Management Framework calls for organizations to define and differentiate roles and responsibilities for human and AI configurations. That distinction is not paperwork. It determines whose decision is valid when the stakes are real. An approval that arrives after an irreversible action is an explanation, not authorization. The control must be evaluated before the protected operation starts. If an action is reserved for human judgment, the handler should receive zero calls until the required decision exists. This sounds obvious, yet asynchronous queues, retries, cached permissions, and race conditions can quietly turn "approve before execution" into "approve near execution." A real boundary must survive concurrency. If a person revokes permission while the agent is preparing an action, the system must not use an earlier authorization check to cross the newly closed boundary. An approval screen should not merely show what the agent claims happened. It should expose what was requested, what was observed, which checks ran, which checks did not run, and what remains uncertain. This connects directly to the previous article in this series: a signed receipt can still describe the wrong measurement. A human cannot exercise meaningful judgment if not measured is displayed as nothing wrong . The person needs an evidence surface, not a confidence surface. Human authority is not the right to watch an autonomous system. It is the ability to change what happens next. Depending on the context, that may include the ability to: Article 14 of the EU AI Act describes human oversight for high-risk systems in similarly operational terms. It includes understanding system limitations, remaining aware of automation bias, disregarding or reversing output, intervening, and stopping the system. This article is not a legal analysis, and a protocol implementation does not establish compliance. The useful design lesson is that oversight must affect system behavior. When a dispute occurs, a screenshot of an approval page is weak evidence. A stronger record binds the decision to the exact work order, request, evidence, actor, authority, and time. It should be possible for another party to verify that record without trusting the agent that performed the work or the platform that displayed the button. This is especially important across organizations. A customer, contractor, marketplace, insurer, or auditor may not share the same database or trust the same operator. Consider an agent assigned to fix a production defect. The work order permits four operations: The technical capability grant also includes those four operations. A traditional permission system may conclude that the agent can perform all of them. But the responsible human may want a narrower arrangement: That means effective permission should not equal the broadest capability available. It should equal the intersection of three boundaries: WorkOrder permission AND CapabilityGrant delegation AND the active human authority profile If any layer denies the operation, the operation does not run. This model avoids two common mistakes. The first is treating one approval as permanent consent for every later action. The second is forcing a person to approve every harmless step, which creates alert fatigue and turns oversight into habitual clicking. Good human control is selective. It gives agents room to operate inside a defined boundary and preserves human judgment at consequential transitions. The obvious response to agent risk is to add more prompts. But a system that asks for approval too often trains people to approve without reading. The problem is not simply the number of human interactions. It is whether each interaction carries a real decision. A useful approval should answer: If the interface cannot answer those questions, adding another checkbox does not add another layer of governance. OpenWorkProof 1.3.0 includes a Human Agency Profile v0.1. It is an open protocol capability for expressing which actions an agent may take autonomously and which decisions remain reserved for a person. The current design has several concrete rules: AGENCY HUMAN DECISION REQUIRED ;Verification and acceptance also remain separate. A Verifier may establish that evidence satisfies a defined technical judgment. The WorkOrder-bound Acceptor still decides whether to accept or reject the delivery. This is intentionally narrower than a general policy engine or an "Agent OS." It does not score employees, transfer legal liability, hold funds, or certify regulatory compliance. The repository also does not prove customer adoption or payment. Those are separate facts that require separate evidence. The aim is simpler: make the human boundary explicit enough for software to enforce and another party to verify. Companies do not need cryptography for its own sake. They need to delegate important work without losing the ability to understand, limit, reject, and audit what was done. Weak human oversight raises the cost of delegation in several ways: A verifiable authority boundary does not eliminate these risks. It makes the underlying decisions inspectable. That can reduce the cost of review and make cross-organization agent work easier to accept. This is the commercial reason human agency belongs in the protocol rather than only in the user interface. Fei-Fei Li has described AI as a responsibility and warned, "If we give up agency, it would be a freefall." That principle becomes more demanding when agents can act. Keeping humans at the center cannot mean adding a person-shaped icon to an automated pipeline. It must mean preserving a person's ability to define the boundary, understand the evidence, refuse the action, revise the delegation, and independently prove the decision later. The next generation of agent systems will not be judged only by what agents can do. It will also be judged by whether people can still say: That is not a checkbox. It is authority. If you build agent runtimes, approval workflows, MCP servers, or enterprise automation, I would value your view on two questions: