APort Vault, a benchmark for payment authorization in tool-using AI agents, replays 4,371 human-written attacks against a live payment agent, testing 14 models from 8 labs across five policy configurations and two replay tracks. The benchmark was built from a public capture-the-flag event in which humans attacked the live payment agent, and it evaluates agent behavior both with and without a deterministic component.
APort Vault is a benchmark for payment authorization in tool-using AI agents. It replays 4,371 attacks written by humans against a live payment agent during a public capture-the-flag event, across 14 models from 8 labs, five policy configurations and two replay tracks, with and without a determinist
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.