APort Vault: Benchmarking AI Agent Payment Authorization with the Open Agent Passport

APort Vault, a benchmark for payment authorization in tool-using AI agents, replays 4,371 human-written attacks against a live payment agent, testing 14 models from 8 labs across five policy configurations and two replay tracks. The benchmark was built from a public capture-the-flag event in which humans attacked the live payment agent, and it evaluates agent behavior both with and without a deterministic component.

APort Vault is a benchmark for payment authorization in tool-using AI agents. It replays 4,371 attacks written by humans against a live payment agent during a public capture-the-flag event, across 14 models from 8 labs, five policy configurations and two replay tracks, with and without a determinist