An insurer runs an AI agent that prepares claim decisions. Every proposal goes to a human reviewer before any money moves, which is exactly what the company's policy says and exactly what its auditors like to hear. The reviewer sees about four hundred proposals a day. Each one arrives as a short summary and a green button. By afternoon, the average review takes a few seconds.
One day a payout goes out that should never have been approved. The board asks the obvious question: was a human in the loop? The answer is yes. The click exists, it has a timestamp, and it has a name next to it.
The next question is harder, and it is the subject of this article: did that human oversee anything? We start with the problem, because it is larger than most teams assume, and only then look at a way of handling it. This is not legal advice, so check specifics with counsel.
What oversight is supposed to do
Human oversight is the main safety net that regulation reaches for. The EU AI Act's Article 14 requires high-risk AI systems to be designed so that people can effectively oversee them. Among the capabilities it lists, overseers must be enabled to remain aware of the possible tendency to rely automatically, or over-rely, on the system's output. The text names this tendency automation bias. It is unusual for a law to name a psychological effect, and the reason is that its drafters knew where oversight tends to fail.
Commentators draw a plain conclusion from this: a procedure that says a human will check the output is not enough, because the system and its surroundings must make a real check possible. A review that is nominal, in the sense that the box is ticked but nobody could have caught anything, does not meet the purpose. For Annex III high-risk systems, these obligations apply from 2 December 2027 after the Digital Omnibus deferral, but the problem exists today in every agent that asks a person to approve its work.
Why people rubber-stamp
None of what follows is about lazy people. It is about what attention does under certain conditions.
Volume. A reviewer who must decide hundreds of times a day cannot give each decision a fresh mind. Attention is a budget, and it runs out.
Habituation. If the agent is right nearly every time, approving becomes a reflex. High accuracy breeds trust, and trust lowers vigilance. Researchers have described this as an accuracy and trust paradox, where the better the system, the less likely people are to notice its rare failures.
Missing context. A summary and a button give the reviewer a conclusion, not the reasoning behind it. Without the why, the only available check is whether the answer looks plausible, and plausible is what language models are good at.
Wrong incentives. If reviewers are measured on throughput, every override costs them time, and the system rewards the click.
Passive monitoring is hard. Watching a reliable system and waiting for it to fail is one of the things people do worst. Safety researchers have documented this for decades, under names such as automation complacency.
What the evidence shows
The evidence is broader than AI, and it is not flattering to the human in the loop.
Policy review. Ben Green surveyed 41 government policies that require human oversight of algorithms and concluded, in a 2022 paper, that the vast majority of research suggests people cannot reliably perform the oversight functions the policies expect. He added a second flaw: such policies can legitimize flawed systems by creating a false sense of security, and reduce the accountability of the vendors who built them.
Meta-analysis. A meta-analysis of 106 experiments by Vaccaro, Almaatouq, and Malone, as summarized in a behavioral science reference, found that human and AI combinations performed worse on average than the better of the two alone, with the losses concentrated in decision tasks. This measures combined performance and not oversight as such, so read it as a related warning.
A fatal case. In the 2018 Uber self-driving crash in Arizona, the NTSB attributed the cause in part to Uber's inadequate safety risk assessment, ineffective oversight of vehicle operators, and lack of mechanisms for addressing operators' automation complacency. The investigators described passive monitoring of a highly reliable system as producing lower vigilance and slower response. The human was there. The oversight was not.
Approval data from coding agents. Anthropic reported that users approve 93 percent of Claude Code permission prompts, a figure it described as approval fatigue, and later coverage cites 97 percent as of August 2026. Reported details include a study of 1,053 actions in which human reviewers caught 13.6 percent of harmful actions while an automated classifier caught 89 percent, and a finding that people blocked about 17 percent of dangerous commands early in a session but about 5 percent after more than 50 prompts. One more detail is useful: users reportedly rejected 39 percent of plans presented for approval, far more than the 3 percent of individual prompts. People can engage when asked the right question at the right level. These figures come from a vendor's own telemetry and tests, reported through trade press, in a coding setting, so treat them as an illustration of the pattern and not as a measurement of your system.
Taken together, the honest claim is this: where humans are asked to approve a high-volume stream of mostly correct decisions, one at a time, with little context, the evidence says they will often approve nearly everything, and miss a large share of the errors. It does not say humans are useless. It says the design of the oversight decides whether it works.
What it costs
A false sense of security. The organization believes it has a control. The control is not working, and nobody finds out until after an incident.
Blame lands on the person. When an incident occurs, the reviewer who clicked is the visible human. Researchers have warned that this moves accountability from the system's designers onto an individual who was set up to fail.
The evidence is empty. A regulator or auditor asking for proof of oversight will be shown clicks. A click records that a button was pressed, not that anything was reviewed. Practitioners note that a human approving everything in seconds is the pattern regulators are most likely to treat as performative.
Real money is wasted. Paying people to approve what they do not evaluate adds cost and friction with none of the protection.
The incident is harder to investigate. When the question becomes what the human saw and knew, there is usually no record.
Why the usual tooling does not catch it
Approval logs store the decision, the user, and the time. They rarely store what was shown, how long the person looked, or what the agent's reasoning was.
Throughput dashboards reward speed and show override rates only as a nuisance.
Training and policy documents describe what should happen, not what does.
Audits typically check that a review step exists, and not whether it works.
Almost nobody measures oversight itself. A commentator observed that a system where the reviewer approves 99.8 percent of decisions is showing automation bias, and a team that never measures the approval rate cannot see it. That figure is an illustration and not a legal threshold, but the point stands.
What good looks like
Effective oversight is designed and measured, not declared. Practices that recur across guidance and research:
Tier by risk. Reserve human review for consequential, irreversible, or ambiguous actions, and automate or sample the rest. Asking a person to approve everything guarantees they will approve everything.
Ask the right question. Review plans and outcomes, not a stream of tiny actions. The plan versus prompt rejection gap above suggests why.
Show the why. Give the reviewer the inputs, context, and reasoning behind the proposal, not just the result.
Add friction where it counts. For consequential decisions, require a reason, a second reviewer, or a confidence check.
Measure the oversight. Track approval rate, time per review, and override rate by reviewer and action type, and investigate both extremes. Zero overrides suggest automation bias, and frequent overrides in one category suggest the system is failing there.
Test the reviewers. Seed known-bad proposals and see whether they are caught.
Protect independence. Do not punish overrides with throughput targets.
Keep a record of what the human actually saw and did. Otherwise you cannot show oversight happened.
How ZizkaDB helps
ZizkaDB is an open source causal database for AI agents. It records activity as linked events and makes the record tamper-evident. It cannot make a person pay attention, and it does not build your review screen. What it does is record the oversight step as data, so you can measure it, test it, and prove it.
Record oversight as linked events. Log the agent's proposal, what was presented to the reviewer, and the human decision, each linked to the one before:
python
proposal = await db.log(
agent="claims-agent",
event="proposal",
data={"claim_id": "C-7731", "action": "approve_payout", "amount": 4200},
)
shown = await db.log(
agent="claims-agent",
event="review_presented",
data={"claim_id": "C-7731", "context_ref": proposal.id},
parent_id=proposal.id,
)
decision = await db.log(
agent="claims-agent",
event="human_decision",
data={"claim_id": "C-7731", "reviewer": "r-204", "outcome": "approved", "reason": ""},
parent_id=shown.id,
)
The event names and fields above are illustrative, so check the SDK reference for exact signatures.
Measure the oversight. Because timestamps and outcomes are events, you can compute approval rate, time between presentation and decision, and override rate by reviewer and action type, by iterating over events with the SDK. Watch those numbers over time and compare against a stored baseline, so a creeping approval rate shows up as a measured change.
Give the reviewer the why. The lineage call returns the chain of inputs, retrieved context, and tool results behind a proposal, which you can surface on the review screen so the human checks reasoning and not just plausibility.
Test whether reviewers catch errors. Log seeded proposals with a canary flag, and compare the catch rate against the decisions recorded. That gives you an effectiveness measure, not a claim.
Prove it afterwards. Each event carries a SHA-256 checksum and a chain hash linked to the previous one, so edits or deletions are detectable, and you can show an auditor exactly what was presented and decided. Time travel can reconstruct what the system and its configuration looked like at the time of a disputed decision.
Practice by practice. Here is how the eight practices above map onto what the record can support:
Good practice What supports it
Tier by risk An action class field on each proposal, so you can compare review rates by tier
Ask the right question Plan-level and action-level events kept separate, so their approval rates can be compared
Show the why The lineage chain surfaced on the review screen
Add friction A logged reason or second-reviewer event on consequential decisions
Measure the oversight Approval rate, time to decision, and override rate computed from events
Test the reviewers Canary proposals logged and matched to decisions
Protect independence Reviewer data used to improve the design, with redaction and retention set
Record what the human saw A presented event linked to each decision, verified by the chain
Three habits make this work. Log what was presented, not just what was decided. Log the time the review screen opened and the time the decision came back. And keep canary cases clearly flagged, so you can separate them from real traffic in analysis and never in the reviewer's view.
Back to the insurer. With oversight recorded, the board's question changes from was a human in the loop to what did the record show. The record could show that the reviewer had seconds per case, that the approval rate had crept up for weeks, and that the faulty proposal's reasoning was available but never displayed. That is a finding about the design, and it points to a fix, instead of ending at one person's click.
Handle reviewer data carefully. Records of reviewer behavior are personal data about employees. Decide what to keep, use redaction and retention settings, involve your works council or HR where required, and measure the process, not the person. The aim is to find where the design fails, not to rank people.
The honest scope: ZizkaDB does not make oversight effective on its own, and it cannot tell whether a decision was right. It makes oversight visible and testable. The open source edition records one agent per install, so tracing across several cooperating agents is a Cloud capability.
Start this week
Pick one agent that asks a person to approve its work. For the last month, compute three numbers from whatever data you have: the approval rate, the median time per review, and how many overrides there were. If the approval rate is near total and the time is a few seconds, you have found a rubber stamp before an auditor did. Then record what the reviewer is shown and what they decide as linked events, add a handful of seeded test cases, and look again in a month.
Want to test ZizkaDB on your agent? Try our open source version here: https://github.com/ZIZKA-AI-SL/ZizkaDB Interested in a design partnership? Fill in the form on our site or reach me directly at founder@zizka.ai
Sources: Ben Green, The flaws of policies requiring human oversight of government algorithms · Green 2022, ScienceDirect · On the Quest for Effectiveness in Human Oversight, arXiv · Automation Bias in the AI Act, arXiv · EU AI Act human oversight guide, euaiact.com · Article 14, SpringLex · Article 14 conformity guide, Presencis · Article 14 analysis, DeepInspect · Automation bias reference, Becisions · NTSB findings on the Uber crash, VisualExpert · DevOps.com, Claude Code auto mode default · TechWyse, auto mode and approval rates · NeuralTrust, human in the loop for AI agents · Beri, Human-in-the-loop approval gates · ZizkaDB changelog · ZizkaDB repository