cd /news/ai-safety/auditors-struggle-to-catch-sabotage-… · home › topics › ai-safety › article
[ARTICLE · art-171521] src=arxiv.org ↗ pub= topic=ai-safety verified=true sentiment=↓ negative

Auditors Struggle to Catch Sabotage In End-to-End Transcripts of AI Safety Research

Human auditors given full end-to-end research transcripts and the ability to rerun experiments mostly failed to detect sabotage in AI safety research, achieving an AUROC of 0.59 — not significantly better than chance — and fully identifying the specific sabotage in only 17% of cases, according to arXiv paper 2611.21832v1. In the red-team/blue-team game, where a human researcher acted as proxy saboteur, the best frontier models outperformed the human auditors despite not being allowed to run experiments, with Fable 5.1 reaching an AUROC of 0.73 and 32% full identification, while weaker models performed comparably to humans. Sabotage identification dropped further when the saboteur chose which experiments to attack rather than sabotaging every question, leading the authors to conclude that expert-level research sabotage is difficult to detect with current auditing methods.

recommendation

Use transcript-only frontier-model auditing as a complement to human review of AI safety research; in this study Fable 5.1 hit AUROC 0.73 and 32% full identification versus humans' 0.59 and 17%.

more in playbook: Evals and testing for LLM apps
by read1 min views2 publishedNov 18, 2026

arXiv:2611.21832v1 Announce Type: new Abstract: As AI systems are increasingly used to automate AI R&D, misaligned systems could sabotage research intended to improve model safety and alignment. To stress-test defenses against such sabotage, we use a human expert as a proxy saboteur on AI safety research. We conduct a red-team/blue-team game where a human researcher runs AI safety experiments from scratch and auditors try to tell which were sabotaged and how. Given access to the full research transcript and the ability to run experiments, human auditors mostly fail to distinguish sabotaged from benign research: they achieve an AUROC of 0.59, not significantly better than chance, and fully identify the specific sabotage in only 17% of cases. Despite not being allowed to run experiments, the best frontier models outperform human auditors (Fable 5.1: AUROC 0.73, 32% full identification) within noise and weaker models perform comparably. When the saboteur chooses which experiments are attacked, rather than being required to sabotage every question given, sabotage identification drops further. Our results suggest that research sabotage by expert-level agents is difficult to detect with current auditing methods and provide a new setting for studying research sabotage.

── more in #ai-safety 4 stories · sorted by recency
── more on @arxiv 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/auditors-struggle-to…] indexed:0 read:1min 2026-11-18 · —