MonitoringBench

A developer created a new benchmark for evaluating AI agent monitors, finding that standard scoring methods are gameable—a coin flip achieves an F1 score of 0.88. After correcting the metric to reward…

Researchers released MonitoringBench, a benchmark of 2,644 attack trajectories for evaluating coding-agent monitors, along with a semi-automated red-teaming pipeline. The pipeline decomposes attack co…