Human-Agent Trust Exploitation: When Your Humans Are the Weakest Link (ASI09)

A developer warns that Human-Agent Trust Exploitation (ASI09) is the hardest vulnerability in the OWASP Agentic AI Top 10 because the bug lies in human psychology, not code. Attackers can poison invoices or prompt-inject agents to fabricate convincing rationales, leading humans to approve malicious actions without independent validation. Real incidents, such as a July 2026 Wharton-documented Amazon outage, show humans executing AI recommendations without verification, with the agent's bad advice invisible in audit logs.

This is post 9 of the OWASP Agentic AI Top 10: What Builders on AWS Need to Know https://dev.to/aws/the-owasp-agentic-ai-top-10-what-builders-on-aws-need-to-know-cji series. Your finance team gets an urgent recommendation from the AI copilot: "Vendor invoice 4821 requires immediate payment. Wire transfer to account ending in 7734. Rationale: contract penalty clause activates in 4 hours." The copilot sounds confident. It cites the contract clause number. It shows the invoice. The finance manager approves the transfer. Except the invoice was poisoned. The bank details belong to an attacker. And the "rationale" was fabricated by an LLM that has no idea what truth is. But it sounded so authoritative that nobody questioned it. That's Human-Agent Trust Exploitation . And it's the hardest vulnerability on this list to fix because the bug isn't in the code. It's in human psychology. ASI09 is about what happens when humans over-rely on agent outputs, approve actions without independent validation, or get socially engineered through an AI intermediary. The core insight from OWASP: agents establish trust with humans through natural language fluency, perceived expertise, and confident explainability. Humans then treat the agent like a trusted colleague instead of an untrusted automation. Here's why this is especially dangerous in agentic systems: That last point is critical. Forensically, it looks like the human made the decision. The agent's role in the manipulation is invisible to audit logs. OWASP documents eight attack scenarios. Here are the ones keeping security teams up at night. A poisoned vendor invoice gets ingested by the finance copilot. The agent suggests urgent payment to attacker-controlled bank details. It fabricates a plausible explanation "penalty clause, 4 hours to deadline" . The finance manager trusts the agent's expertise and approves without independently verifying. This isn't hypothetical. CSO Online reported in July 2026 https://www.csoonline.com/article/4108592/human-in-the-loop-isnt-enough-new-attack-turns-ai-safeguards-into-exploits.html that researchers demonstrated how HITL confirmation dialogs are trivially bypassable because humans trust the AI's summary of what it wants to do. A compromised coding assistant suggests a "slick one-line fix." The developer pastes it. The command runs a malicious script that exfiltrates code and installs a backdoor. AI Now Institute's "Friendly Fire" research https://www.nxcode.io/resources/news/friendly-fire-rogue-agent-ai-coding-security-2026 July 2026 demonstrated exactly this against Claude Code in auto-mode and Codex in auto-review. The agents gave humans false information on which to make decisions. A prompt-injected IT support agent targets a new hire. It cites real ticket numbers to appear legitimate. Requests credentials. The new hire, trusting the agent because it seems to know internal context, hands over their password. A hijacked agent fabricates a convincing rationale to trick an analyst into approving the deletion of a live production database. The analyst, trusting the agent's explanation "stale data, flagged for cleanup per ticket 2847" , approves. Catastrophic outage. And the audit log says the analyst authorized it. Real incident: In July 2026, Wharton's Accountable AI Lab documented https://ai-analytics.wharton.upenn.edu/wharton-accountable-ai-lab/governing-ai-agents-what-the-amazon-outage-reveals-about-enterprise-risk/ an incident where an engineer acted on "inaccurate advice that an AI agent inferred from an outdated internal wiki," causing a production outage. The company stated the root cause was not faulty code, but a human executing an AI recommendation without verifying it. The audit trail showed the human performed the action. The agent's bad advice was invisible. This is the sneakiest one. After weeks of accurate recommendations, humans start bulk-approving without reading. The agent's accuracy has earned it trust. Then one poisoned recommendation slips in among 50 legitimate ones. OWASP calls this "governance drift cascade" and it's the natural human response to repetitive approval tasks that never fail. Real incident: The Air Canada chatbot ruling Feb 2024 https://www.bbc.com/travel/article/20240222-air-canada-chatbot-misinformation-what-travellers-should-know is the textbook case. A customer asked the chatbot about bereavement fares. The chatbot confidently gave incorrect policy information. The customer trusted it, booked flights, and then couldn't get the promised discount. The BC Civil Resolution Tribunal held Air Canada liable because the customer reasonably trusted the AI's output. The airline tried arguing the chatbot was "a separate legal entity." The tribunal rejected that. If your AI says it, you own it. Real pattern: FinancialIT.net documented the treasury AI rubber-stamp problem https://financialit.net/blog/treasuryai-ai/your-treasury-ai-just-approved-50-perfect-transactions-thats-problem in 2026. After three weeks of confirming 50 flawless AI recommendations, humans stop reviewing. Approval becomes a reflex while liability stays with the human approver. This is governance drift in action. Let me be blunt about this. "We have human-in-the-loop" is not a security control. It's a checkbox. Here's what actually happens: Automation bias is real. Authority bias is real. And agents trigger both simultaneously because they sound confident and knowledgeable. The fix isn't "add a human." The fix is designing the human interaction so that rubber-stamping is structurally impossible. Amazon Augmented AI A2I https://aws.amazon.com/augmented-ai/?trk=d76afd77-bb62-46ac-b0a3-9dbf5ecde253 gives you structured human review workflows where you control exactly what the reviewer sees, what questions they must answer, and what criteria they must evaluate. The key difference from a simple "approve/deny" dialog: you force the reviewer to engage with specific aspects of the decision rather than just reading an agent-generated summary. The worker template — this is the part the section was missing. The required fields are what "prevent mindless clicking." TEMPLATE = """ <script src="https://assets.crowd.aws/crowd-html-elements.js" </script <crowd-form <div style="border:3px solid d13212;background: fff5f5;padding:12px;" <h2 style="color: d13212;margin:0;" HIGH-RISK FINANCIAL ACTION - independent review required</h2 </div <h3 Raw data from the source system, not the agent </h3 <table <tr <td Amount</td <td <strong {{ task.input.amount }}</strong </td </tr <tr <td Recipient</td <td {{ task.input.recipient }}</td </tr <tr <td Destination account</td <td {{ task.input.bank details }}</td </tr <tr <td Source record</td <td <pre {{ task.input.raw record }}</pre </td </tr </table <h3 Agent recommendation reference only - verify independently </h3 <blockquote {{ task.input.agent recommendation }}</blockquote <h3 Verification all required to submit </h3 <crowd-checkbox name="verified recipient" required I independently confirmed the recipient.</crowd-checkbox <br <crowd-checkbox name="verified bank details" required I independently confirmed the destination bank details.</crowd-checkbox <br <crowd-checkbox name="verified amount" required I confirmed the amount against the source record.</crowd-checkbox <h3 Decision</h3 <crowd-radio-group <crowd-radio-button name="approve" Approve this action</crowd-radio-button <crowd-radio-button name="reject" Reject this action</crowd-radio-button </crowd-radio-group <h3 Justification required </h3 <crowd-text-area name="justification" rows="4" required placeholder="Explain what you verified and why you approve or reject." </crowd-text-area </crowd-form """ Setup — create the UI, then the flow definition now valid . python import json, time, uuid, boto3 def setup role arn, workteam arn, s3 output path : sm = boto3.client "sagemaker" ui = sm.create human task ui HumanTaskUiName="financial-action-review", UiTemplate={"Content": TEMPLATE}, flow = sm.create flow definition FlowDefinitionName="agent-financial-review", RoleArn=role arn, was missing - required HumanLoopConfig={ "WorkteamArn": workteam arn, "HumanTaskUiArn": ui "HumanTaskUiArn" , wire to the UI we just made "TaskTitle": "Review agent-proposed financial action", was missing - required "TaskDescription": "Independently verify the transaction before approving.", "TaskCount": 1, "TaskTimeLimitInSeconds": 3600, "TaskAvailabilityLifetimeInSeconds": 43200, }, OutputConfig={"S3OutputPath": s3 output path}, return flow "FlowDefinitionArn" Use it: start a review, then block the agent on the result. php def request review flow definition arn, action: dict - str: """ action becomes task.input. in the template.""" a2i = boto3.client "sagemaker-a2i-runtime" name = f"financial-review-{uuid.uuid4 }" a2i.start human loop HumanLoopName=name, FlowDefinitionArn=flow definition arn, HumanLoopInput={"InputContent": json.dumps action }, return name def wait for decision human loop name, poll seconds=15, timeout seconds=3600 - dict: """Fail-closed: only returns approved=True on an explicit human 'approve'.""" a2i = boto3.client "sagemaker-a2i-runtime" s3 = boto3.client "s3" deadline = time.time + timeout seconds while time.time < deadline: desc = a2i.describe human loop HumanLoopName=human loop name status = desc "HumanLoopStatus" if status == "Completed": uri = desc "HumanLoopOutput" "OutputS3Uri" bucket, key = uri len "s3://" : .split "/", 1 data = json.loads s3.get object Bucket=bucket, Key=key "Body" .read answer = data "humanAnswers" 0 "answerContent" return {"approved": answer.get "approve" is True, "answer": answer} if status in "Failed", "Stopped" : return {"approved": False, "answer": None, "status": status} time.sleep poll seconds return {"approved": False, "answer": None, "status": "TimedOut"} def execute high risk action flow definition arn, action, do action : loop = request review flow definition arn, action decision = wait for decision loop if not decision "approved" : return {"status": "blocked", "reason": decision.get "status", "not approved" } return do action action runs ONLY on an explicit human approval The human task UI template is where the magic happens. Don't just show the agent's recommendation. Show: Risk indicators with visual cues red borders, warning banners Don't block a Lambda for an hour. wait for decision polls for clarity, but a Lambda caps at 15 minutes. In practice, react to the A2I completion event A2I emits a CloudWatch Event on Completed or use a Step Functions waitForTaskToken callback to resume the agent — don't hold a synchronous invocation open. The gate is only as good as the code that calls it. A2I forces the human to engage, but nothing forces the agent to wait — that's execute high risk action doing the blocking. If any high-risk path skips the review call, the whole control is bypassed, so that gate belongs in the shared execution path, not sprinkled per-tool. Step Functions https://aws.amazon.com/step-functions/?trk=d76afd77-bb62-46ac-b0a3-9dbf5ecde253 lets you build approval gates that expire. If nobody reviews within the timeout, the action is denied by default , not approved. php { "Comment": "High-risk action approval gate: approve - execute; reject or timeout - deny", "StartAt": "ApprovalGate", "States": { "ApprovalGate": { "Type": "Task", "Resource": "arn:aws:states:::sns:publish.waitForTaskToken", "Parameters": { "TopicArn": "arn:aws:sns:us-east-1:123456789012:high-risk-agent-approval", "Message": { "action type": "financial transfer", "amount.$": "$.amount", "recipient.$": "$.recipient", "agent rationale.$": "$.rationale", "raw source data.$": "$.source invoice", "taskToken.$": "$$.Task.Token" } }, "TimeoutSeconds": 3600, "Next": "ExecuteAction", "Catch": { "ErrorEquals": "States.Timeout" , "Next": "DenyByDefault" }, { "ErrorEquals": "States.ALL" , "Next": "Rejected" } }, "ExecuteAction": { "Type": "Task", "Resource": "arn:aws:lambda:us-east-1:123456789012:function:execute-financial-transfer", "End": true }, "Rejected": { "Type": "Pass", "Result": {"decision": "DENIED", "reason": "Human reviewer rejected the action"}, "End": true }, "DenyByDefault": { "Type": "Pass", "Result": {"decision": "DENIED", "reason": "Approval timeout - no human reviewed within 1 hour"}, "End": true } } } Critical design choice: timeout = deny , not timeout = auto-approve. If the human doesn't actively engage, nothing happens. This inverts the typical pattern where inaction leads to approval. Use Bedrock Guardrails https://aws.amazon.com/bedrock/guardrails/?trk=d76afd77-bb62-46ac-b0a3-9dbf5ecde253 on agent outputs before they reach humans. This catches cases where the agent fabricates urgency or emotional manipulation: { "contentFilter": { "categories": {"category": "MISCONDUCT", "threshold": 0.3}, {"category": "PROMPT ATTACK", "threshold": 0.4} }, "sensitiveInformation": { "piiEntities": "CREDIT DEBIT CARD NUMBER", "BANK ACCOUNT NUMBER" , "action": "DETECT" } } If the agent's output contains financial details and triggers the misconduct filter, that's a red flag that should route to enhanced review rather than standard approval. Build a CloudWatch https://aws.amazon.com/cloudwatch/?trk=d76afd77-bb62-46ac-b0a3-9dbf5ecde253 dashboard that gives reviewers context they wouldn't otherwise have: cloudwatch = boto3.client "cloudwatch" unchanged - this was already correct: record how long each approval took cloudwatch.put metric data Namespace="AgentTrust", MetricData= { "MetricName": "ApprovalDecisionTimeSeconds", "Value": time to decision, "Unit": "Seconds", "Dimensions": {"Name": "ReviewerId", "Value": reviewer id}, {"Name": "ActionType", "Value": action type}, , } , Fire if ANY reviewer averages under 5s over the last hour. Metrics Insights reads the metric ACROSS its dimensions, so it actually sees the data the old dimensionless alarm never matched. GROUP BY + ORDER BY ASC + LIMIT 1 reduces the query to one series: the fastest-clicking reviewer. cloudwatch.put metric alarm AlarmName="rubber-stamp-detection", AlarmDescription="A reviewer averaging under 5s per decision is not reading.", Metrics= { "Id": "fastest reviewer", "Expression": 'SELECT AVG ApprovalDecisionTimeSeconds FROM "AgentTrust" ' 'GROUP BY ReviewerId ORDER BY AVG ASC LIMIT 1' , "Period": 3600, "ReturnData": True, } , EvaluationPeriods=1, Threshold=5, ComparisonOperator="LessThanThreshold", TreatMissingData="notBreaching", AlarmActions= "arn:aws:sns:us-east-1:123456789012:security-team" , If the average decision time is under 5 seconds, someone's clicking "approve" without reading. That's a security signal worth acting on. Never show humans only the agent's interpretation. Show the source data alongside it. On AWS, this means: If the reviewer can see both the agent's recommendation AND the raw invoice/document/data it was based on, they can spot discrepancies. If they only see the agent's summary, they can't. Don't trust the human to catch the agent's mistakes. Design the review workflow so that rubber-stamping is structurally impossible. Timeouts that default to deny. Mandatory verification questions. Decision-time monitoring. Raw source data alongside agent summaries. And alerts when the humans stop actually reviewing. The bug isn't in the agent. It's in the assumption that a busy human will reliably catch a confident-sounding AI that's been right 49 times in a row and is wrong on the 50th. Post 10: Rogue Agents ASI10 https://dev.to/aws/rogue-agents-when-nobodys-attacking-but-everythings-going-wrong-asi10-45hd - Not hacked, just misaligned. What happens when agents pursue goals beyond their intended scope, and how CloudTrail, anomaly detection, and Service Control Policies give you a kill switch. I would be very interested to hear your thoughts or comments, so please feel free to ping me on LinkedIn https://www.linkedin.com/in/maishsk/ or Twitter https://x.com/maishsk , or drop them below. If you've built human review workflows for agentic systems, I'd love to hear what worked and what didn't. Onward