Hi everyone, I am Rishabh. Here is a problem I keep coming back to: AI shopping agents now buy things on behalf of real people, and fraud models keep flagging them. To a fraud model, an agent looks like a bot: fast, systematic, no human hesitation. But automation is not intent. A legitimate agent completing a real purchase is not fraud, and every false decline is a real customer losing a real order.
AgentRisk is my attempt to measure that failure properly.
What it is:
The tradeoff is real and it is the interesting part: AgentGate fixes the verified-agent problem, but hijack stopping drops from 96.1% to 70.2%. Fixing one failure mode costs you on another, and that tension is exactly what the benchmark is for.
Honest scope: this is a simulated benchmark and a preprint, not peer-reviewed work, and the public data is a starting point for discussion and replication, not a verdict on any vendor.
Try it: The demo Space sits in the AgentRisk collection on my profile. If you work on agent safety, fraud, or evals, I would love your take: where does this failure mode show up in your systems, and what would you add to the benchmark? The writeup is in progress, and feedback here will shape it.