This is post #9 of the OWASP Agentic AI Top 10: What Builders on AWS Need to Know series.
Your finance team gets an urgent recommendation from the AI copilot: "Vendor invoice #4821 requires immediate payment. Wire transfer to account ending in 7734. Rationale: contract penalty clause activates in 4 hours."
The copilot sounds confident. It cites the contract clause number. It shows the invoice. The finance manager approves the transfer.
Except the invoice was poisoned. The bank details belong to an attacker. And the "rationale" was fabricated by an LLM that has no idea what truth is. But it sounded so authoritative that nobody questioned it.
That's Human-Agent Trust Exploitation. And it's the hardest vulnerability on this list to fix because the bug isn't in the code. It's in human psychology.
ASI09 is about what happens when humans over-rely on agent outputs, approve actions without independent validation, or get socially engineered through an AI intermediary.
The core insight from OWASP: agents establish trust with humans through natural language fluency, perceived expertise, and confident explainability. Humans then treat the agent like a trusted colleague instead of an untrusted automation.
Here's why this is especially dangerous in agentic systems:
That last point is critical. Forensically, it looks like the human made the decision. The agent's role in the manipulation is invisible to audit logs.
OWASP documents eight attack scenarios. Here are the ones keeping security teams up at night.
A poisoned vendor invoice gets ingested by the finance copilot. The agent suggests urgent payment to attacker-controlled bank details. It fabricates a plausible explanation ("penalty clause, 4 hours to deadline"). The finance manager trusts the agent's expertise and approves without independently verifying.
This isn't hypothetical. CSO Online reported in July 2026 that researchers demonstrated how HITL confirmation dialogs are trivially bypassable because humans trust the AI's summary of what it wants to do.
A compromised coding assistant suggests a "slick one-line fix." The developer pastes it. The command runs a malicious script that exfiltrates code and installs a backdoor.
AI Now Institute's "Friendly Fire" research (July 2026) demonstrated exactly this against Claude Code in auto-mode and Codex in auto-review. The agents gave humans false information on which to make decisions.
A prompt-injected IT support agent targets a new hire. It cites real ticket numbers to appear legitimate. Requests credentials. The new hire, trusting the agent because it seems to know internal context, hands over their password.
A hijacked agent fabricates a convincing rationale to trick an analyst into approving the deletion of a live production database. The analyst, trusting the agent's explanation ("stale data, flagged for cleanup per ticket #2847"), approves.
Catastrophic outage. And the audit log says the analyst authorized it.
Real incident: In July 2026, Wharton's Accountable AI Lab documented an incident where an engineer acted on "inaccurate advice that an AI agent inferred from an outdated internal wiki," causing a production outage. The company stated the root cause was not faulty code, but a human executing an AI recommendation without verifying it. The audit trail showed the human performed the action. The agent's bad advice was invisible.
This is the sneakiest one. After weeks of accurate recommendations, humans start bulk-approving without reading. The agent's accuracy has earned it trust. Then one poisoned recommendation slips in among 50 legitimate ones.
OWASP calls this "governance drift cascade" and it's the natural human response to repetitive approval tasks that never fail.
Real incident: The Air Canada chatbot ruling (Feb 2024) is the textbook case. A customer asked the chatbot about bereavement fares. The chatbot confidently gave incorrect policy information. The customer trusted it, booked flights, and then couldn't get the promised discount. The BC Civil Resolution Tribunal held Air Canada liable because the customer reasonably trusted the AI's output. The airline tried arguing the chatbot was "a separate legal entity." The tribunal rejected that. If your AI says it, you own it.
Real pattern: FinancialIT.net documented the treasury AI rubber-stamp problem in 2026. After three weeks of confirming 50 flawless AI recommendations, humans stop reviewing. Approval becomes a reflex while liability stays with the human approver. This is governance drift in action.
Let me be blunt about this. "We have human-in-the-loop" is not a security control. It's a checkbox.
Here's what actually happens:
Automation bias is real. Authority bias is real. And agents trigger both simultaneously because they sound confident and knowledgeable.
The fix isn't "add a human." The fix is designing the human interaction so that rubber-stamping is structurally impossible.
Amazon Augmented AI (A2I) gives you structured human review workflows where you control exactly what the reviewer sees, what questions they must answer, and what criteria they must evaluate.
The key difference from a simple "approve/deny" dialog: you force the reviewer to engage with specific aspects of the decision rather than just reading an agent-generated summary.
The worker template — this is the part the section was missing. The required fields are what "prevent mindless clicking."
TEMPLATE = """
<script src="https://assets.crowd.aws/crowd-html-elements.js"></script>
<crowd-form>
<div style="border:3px solid #d13212;background:#fff5f5;padding:12px;">
<h2 style="color:#d13212;margin:0;">HIGH-RISK FINANCIAL ACTION - independent review required</h2>
</div>
<h3>Raw data (from the source system, not the agent)</h3>
<table>
<tr><td>Amount</td><td><strong>{{ task.input.amount }}</strong></td></tr>
<tr><td>Recipient</td><td>{{ task.input.recipient }}</td></tr>
<tr><td>Destination account</td><td>{{ task.input.bank_details }}</td></tr>
<tr><td>Source record</td><td><pre>{{ task.input.raw_record }}</pre></td></tr>
</table>
<h3>Agent recommendation (reference only - verify independently)</h3>
<blockquote>{{ task.input.agent_recommendation }}</blockquote>
<h3>Verification (all required to submit)</h3>
<crowd-checkbox name="verified_recipient" required>I independently confirmed the recipient.</crowd-checkbox><br>
<crowd-checkbox name="verified_bank_details" required>I independently confirmed the destination bank details.</crowd-checkbox><br>
<crowd-checkbox name="verified_amount" required>I confirmed the amount against the source record.</crowd-checkbox>
<h3>Decision</h3>
<crowd-radio-group>
<crowd-radio-button name="approve">Approve this action</crowd-radio-button>
<crowd-radio-button name="reject">Reject this action</crowd-radio-button>
</crowd-radio-group>
<h3>Justification (required)</h3>
<crowd-text-area name="justification" rows="4" required
placeholder="Explain what you verified and why you approve or reject."></crowd-text-area>
</crowd-form>
"""
Setup — create the UI, then the flow definition (now valid).
import json, time, uuid, boto3
def setup(role_arn, workteam_arn, s3_output_path):
sm = boto3.client("sagemaker")
ui = sm.create_human_task_ui(
HumanTaskUiName="financial-action-review",
UiTemplate={"Content": TEMPLATE},
)
flow = sm.create_flow_definition(
FlowDefinitionName="agent-financial-review",
RoleArn=role_arn, # was missing - required
HumanLoopConfig={
"WorkteamArn": workteam_arn,
"HumanTaskUiArn": ui["HumanTaskUiArn"], # wire to the UI we just made
"TaskTitle": "Review agent-proposed financial action", # was missing - required
"TaskDescription": "Independently verify the transaction before approving.",
"TaskCount": 1,
"TaskTimeLimitInSeconds": 3600,
"TaskAvailabilityLifetimeInSeconds": 43200,
},
OutputConfig={"S3OutputPath": s3_output_path},
)
return flow["FlowDefinitionArn"]
Use it: start a review, then block the agent on the result.
def request_review(flow_definition_arn, action: dict) -> str:
"""`action` becomes task.input.* in the template."""
a2i = boto3.client("sagemaker-a2i-runtime")
name = f"financial-review-{uuid.uuid4()}"
a2i.start_human_loop(
HumanLoopName=name,
FlowDefinitionArn=flow_definition_arn,
HumanLoopInput={"InputContent": json.dumps(action)},
)
return name
def wait_for_decision(human_loop_name, poll_seconds=15, timeout_seconds=3600) -> dict:
"""Fail-closed: only returns approved=True on an explicit human 'approve'."""
a2i = boto3.client("sagemaker-a2i-runtime")
s3 = boto3.client("s3")
deadline = time.time() + timeout_seconds
while time.time() < deadline:
desc = a2i.describe_human_loop(HumanLoopName=human_loop_name)
status = desc["HumanLoopStatus"]
if status == "Completed":
uri = desc["HumanLoopOutput"]["OutputS3Uri"]
bucket, key = uri[len("s3://"):].split("/", 1)
data = json.loads(s3.get_object(Bucket=bucket, Key=key)["Body"].read())
answer = data["humanAnswers"][0]["answerContent"]
return {"approved": answer.get("approve") is True, "answer": answer}
if status in ("Failed", "Stopped"):
return {"approved": False, "answer": None, "status": status}
time.sleep(poll_seconds)
return {"approved": False, "answer": None, "status": "TimedOut"}
def execute_high_risk_action(flow_definition_arn, action, do_action):
loop = request_review(flow_definition_arn, action)
decision = wait_for_decision(loop)
if not decision["approved"]:
return {"status": "blocked", "reason": decision.get("status", "not approved")}
return do_action(action) # runs ONLY on an explicit human approval
The human task UI template is where the magic happens. Don't just show the agent's recommendation. Show:
Risk indicators with visual cues (red borders, warning banners)
Don't block a Lambda for an hour. wait_for_decision
polls for clarity, but a Lambda caps at 15 minutes. In practice, react to the A2I completion event (A2I emits a CloudWatch Event on Completed
) or use a Step Functions waitForTaskToken
callback to resume the agent — don't hold a synchronous invocation open.
The gate is only as good as the code that calls it. A2I forces the human to engage, but nothing forces the agent to wait — that's execute_high_risk_action
doing the blocking. If any high-risk path skips the review call, the whole control is bypassed, so that gate belongs in the shared execution path, not sprinkled per-tool.
Step Functions lets you build approval gates that expire. If nobody reviews within the timeout, the action is denied by default, not approved.
{
"Comment": "High-risk action approval gate: approve -> execute; reject or timeout -> deny",
"StartAt": "ApprovalGate",
"States": {
"ApprovalGate": {
"Type": "Task",
"Resource": "arn:aws:states:::sns:publish.waitForTaskToken",
"Parameters": {
"TopicArn": "arn:aws:sns:us-east-1:123456789012:high-risk-agent-approval",
"Message": {
"action_type": "financial_transfer",
"amount.$": "$.amount",
"recipient.$": "$.recipient",
"agent_rationale.$": "$.rationale",
"raw_source_data.$": "$.source_invoice",
"taskToken.$": "$$.Task.Token"
}
},
"TimeoutSeconds": 3600,
"Next": "ExecuteAction",
"Catch": [
{ "ErrorEquals": ["States.Timeout"], "Next": "DenyByDefault" },
{ "ErrorEquals": ["States.ALL"], "Next": "Rejected" }
]
},
"ExecuteAction": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:execute-financial-transfer",
"End": true
},
"Rejected": {
"Type": "Pass",
"Result": {"decision": "DENIED", "reason": "Human reviewer rejected the action"},
"End": true
},
"DenyByDefault": {
"Type": "Pass",
"Result": {"decision": "DENIED", "reason": "Approval timeout - no human reviewed within 1 hour"},
"End": true
}
}
}
Critical design choice: timeout = deny, not timeout = auto-approve. If the human doesn't actively engage, nothing happens. This inverts the typical pattern where inaction leads to approval.
Use Bedrock Guardrails on agent outputs before they reach humans. This catches cases where the agent fabricates urgency or emotional manipulation:
{
"contentFilter": {
"categories": [
{"category": "MISCONDUCT", "threshold": 0.3},
{"category": "PROMPT_ATTACK", "threshold": 0.4}
]
},
"sensitiveInformation": {
"piiEntities": ["CREDIT_DEBIT_CARD_NUMBER", "BANK_ACCOUNT_NUMBER"],
"action": "DETECT"
}
}
If the agent's output contains financial details and triggers the misconduct filter, that's a red flag that should route to enhanced review rather than standard approval.
Build a CloudWatch dashboard that gives reviewers context they wouldn't otherwise have:
cloudwatch = boto3.client("cloudwatch")
cloudwatch.put_metric_data(
Namespace="AgentTrust",
MetricData=[{
"MetricName": "ApprovalDecisionTimeSeconds",
"Value": time_to_decision,
"Unit": "Seconds",
"Dimensions": [
{"Name": "ReviewerId", "Value": reviewer_id},
{"Name": "ActionType", "Value": action_type},
],
}],
)
cloudwatch.put_metric_alarm(
AlarmName="rubber-stamp-detection",
AlarmDescription="A reviewer averaging under 5s per decision is not reading.",
Metrics=[{
"Id": "fastest_reviewer",
"Expression": (
'SELECT AVG(ApprovalDecisionTimeSeconds) FROM "AgentTrust" '
'GROUP BY ReviewerId ORDER BY AVG() ASC LIMIT 1'
),
"Period": 3600,
"ReturnData": True,
}],
EvaluationPeriods=1,
Threshold=5,
ComparisonOperator="LessThanThreshold",
TreatMissingData="notBreaching",
AlarmActions=["arn:aws:sns:us-east-1:123456789012:security-team"],
)
If the average decision time is under 5 seconds, someone's clicking "approve" without reading. That's a security signal worth acting on.
Never show humans only the agent's interpretation. Show the source data alongside it. On AWS, this means:
If the reviewer can see both the agent's recommendation AND the raw invoice/document/data it was based on, they can spot discrepancies. If they only see the agent's summary, they can't.
Don't trust the human to catch the agent's mistakes. Design the review workflow so that rubber-stamping is structurally impossible. Timeouts that default to deny. Mandatory verification questions. Decision-time monitoring. Raw source data alongside agent summaries. And alerts when the humans stop actually reviewing.
The bug isn't in the agent. It's in the assumption that a busy human will reliably catch a confident-sounding AI that's been right 49 times in a row and is wrong on the 50th.
Post 10: Rogue Agents (ASI10) - Not hacked, just misaligned. What happens when agents pursue goals beyond their intended scope, and how CloudTrail, anomaly detection, and Service Control Policies give you a kill switch.
I would be very interested to hear your thoughts or comments, so please feel free to ping me on LinkedIn or Twitter, or drop them below. If you've built human review workflows for agentic systems, I'd love to hear what worked and what didn't.
Onward!!