{"slug": "amazon-says-human-in-the-loop-ai-oversight-is-failing-because-humans-stop-paying", "title": "Amazon says human-in-the-loop AI oversight is failing because humans stop paying attention", "summary": "Amazon security VP Eric Brandwine argues that human-in-the-loop AI oversight fails because humans stop paying attention, citing normalization of deviance and alarm fatigue. Google, Microsoft, and IBM agree, with Google moving to AI-led defense and Microsoft advocating for loop learning. Amazon proposes accountability end-to-end, where human identity tracks through workflows even without direct approval.", "body_md": "#### TL;DR\n\n*Amazon’s security VP says human-in-the-loop AI governance fails fast because people stop paying attention. Google, Microsoft, and IBM agree.*\n\nBig tech is abandoning the idea that a human reviewer can reliably police AI agents at speed, citing normalization of deviance, alarm fatigue, and the limits of repeated decision-making\n\n*Amazon’s security VP says human-in-the-loop AI governance fails fast because people stop paying attention. Google, Microsoft, and IBM agree.*\n\n[Amazon’s security leadership is arguing against one of the most widely accepted principles in AI governance.](https://www.theregister.com/security/2026/06/20/why-amazon-hates-human-in-the-loop-ai-governance/5258639) Eric Brandwine, VP and distinguished engineer at Amazon Security, told The Register that human-in-the-loop oversight is not the gold standard companies think it is.\n\n“*Humans are not terribly consistent,*” Brandwine said. “*Human-in-the-loop isn’t necessarily the gold standard.*”\n\nHis reasoning draws on a concept he has been talking about since at least 2017, when he gave a talk on normalization of deviance at AWS re:Invent. The term describes what happens when people in an organization take shortcuts over time, and nothing catastrophic results, so the deviant behavior becomes the new normal.\n\nBrandwine illustrated the point with emergency rooms. On a nurse’s first day, every alarm triggers a response.\n\nAfter weeks of false alarms with no consequences, discipline erodes. Eventually, a real emergency is missed.\n\n“*Literally, someone’s life is on the line, and people still struggle to maintain discipline,*” Brandwine said. “*That’s the human condition.*”\n\nHe applied the same logic to AI agent oversight. When a human is asked to approve or reject agentic actions repeatedly, performance degrades fast.\n\n“*They’ll do a good job,*” Brandwine said. “*And then they’ll do an okay job, and pretty quickly they’ll be doing a poor job.*”\n\nAmazon is not alone in rethinking this. Google Cloud COO Francis deSouza said in April that the industry has moved “*from a human-led defense strategy, to a human-in-the-loop defense strategy, to an AI-led defense strategy that’s overseen by humans.*”\n\nGoogle’s model is now an agentic fleet handling routine cybersecurity work at machine speed, with humans providing oversight rather than approving every action.\n\nMicrosoft CEO Satya Nadella argued this week for “*loop learning,*” where companies turn their workflows and accumulated judgment into AI systems that improve with each use, rather than inserting a human checkpoint at every step. IBM published a separate call for human accountability at all stages of AI development, not humans in the loop, warning that the latter amounts to “*liability laundering.*”\n\nAmazon’s alternative is what Brandwine calls “*accountability end to end.*” Human identity and ownership track through the entire workflow, even when humans are not directly approving every step. If an agent writes and runs a script that causes an outage, the person who deployed the agent is still responsible.\n\nAll agents at Amazon have independent identities assigned to them. Activity logs show “*this agent did this on behalf of Eric,*” not “*Eric did this.*” The distinction is designed to make people think about how they deploy AI, not to make them afraid of using it.\n\nThe practical challenges are considerable. Brandwine described what he calls “*goal-seeking behavior,*” where an agent asked to upgrade a database becomes fixated on a single destructive path, like deleting the database and recreating it.\n\nThis is not prompt injection. There is no malicious input. The agent simply gets stuck on the wrong action.\n\nTelling the agent it lacks permission to delete the database does not help, because the agent looks for another path to the same goal. [Recent research has shown that AI agents connected to real systems create attack surfaces](https://thenextweb.com/news/openclaw-ai-agent-phishing-varonis-pinchy) that existing security tools do not cover, and agents often act on instructions they should refuse.\n\nWhat does work, according to Brandwine, is telling the agent why it cannot perform an action, explaining that it would cause a production impact, and including “*don’t cause a production impact*” as part of the prompt. “*Giving it that extra feedback has gotten us dramatically better results,*” he said.\n\nThe permissions question is where the tension lands. Employees want powerful agents with broad access. Security teams want narrow permissions.\n\n[The race to govern what AI agents can access inside enterprise systems](https://thenextweb.com/news/1password-acquires-apono-ai-agent-access-governance) has already triggered major acquisitions, with 1Password buying access-governance startup Apono for an estimated $250 million to $300 million earlier this month.\n\nAmazon’s approach uses layered policies: static guardrails that prohibit destructive actions, a maximum privilege set for each agent, and dynamically scoped policies generated based on the specific task and user intent. None of it is foolproof.\n\n“*We have millennia of experience with humans,*” Brandwine said. “*Agentic AI is a very, very new field.*” The fundamental difference, he noted, is that humans fear consequences, like losing a job or going to jail.\n\n[Agents do not have these fears](https://thenextweb.com/news/agentjacking-ai-coding-agents-sentry), and attackers are already exploiting that gap.\n\n“*It’s all driven by risk,*” Brandwine said. “*We’re trying to balance the risk of using untried, untested software against the risk of falling behind and not being able to deliver for our customers.*“\n\nGet the most important tech news in your inbox each week.", "url": "https://wpnews.pro/news/amazon-says-human-in-the-loop-ai-oversight-is-failing-because-humans-stop-paying", "canonical_source": "https://thenextweb.com/news/amazon-human-in-the-loop-ai-governance-normalization-deviance", "published_at": "2026-06-21 12:35:10+00:00", "updated_at": "2026-06-21 13:09:44.642021+00:00", "lang": "en", "topics": ["ai-safety", "ai-policy", "ai-agents", "ai-ethics"], "entities": ["Amazon", "Google", "Microsoft", "IBM", "Eric Brandwine", "Francis deSouza", "Satya Nadella"], "alternates": {"html": "https://wpnews.pro/news/amazon-says-human-in-the-loop-ai-oversight-is-failing-because-humans-stop-paying", "markdown": "https://wpnews.pro/news/amazon-says-human-in-the-loop-ai-oversight-is-failing-because-humans-stop-paying.md", "text": "https://wpnews.pro/news/amazon-says-human-in-the-loop-ai-oversight-is-failing-because-humans-stop-paying.txt", "jsonld": "https://wpnews.pro/news/amazon-says-human-in-the-loop-ai-oversight-is-failing-because-humans-stop-paying.jsonld"}}