CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat

A benchmark called CyberArena has been introduced to evaluate AI agents in live, symmetric attack-and-defense cybersecurity combat, addressing a gap the authors identify in existing benchmarks that test agents in isolation against static targets. The benchmark is built on the premise that real cyber operations are dynamic and coupled, with security postures constantly changing as attackers and defenders act against each other. The work targets the evaluation of AI agents' offensive and defensive security capabilities as those capabilities strengthen in real-world settings.

AI agents are demonstrating increasingly strong offensive and defensive cybersecurity capabilities in the real world. Current benchmarks mostly evaluate agents in isolation against static targets. Real cyber operations, however, are dynamic and coupled: security postures constantly change as attacke