Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits

Researchers identified five failure modes in perturbation-based construct-validity audits of AI safety benchmarks, demonstrating that implementation details can silently invalidate audit conclusions. In a case study of two open-weight instruction-tuned models across five safety benchmarks, every audit failed to meet a proposed six-point due-diligence gate, highlighting the fragility of current audit practices.

arXiv:2607.02586v1 Announce Type: new Abstract: Governance frameworks ask AI providers and auditors for documented evaluation evidence, and perturbation-based construct-validity audits are a common form of that evidence. We argue the audits are themselves fragile: their conclusions can be silently manufactured by implementation details that readers cannot see in the reported numbers. We name five classes of pipeline failure and demonstrate each in a self-audit over safety benchmarks and open-weight instruction-tuned models. Under a unified six-point due-diligence gate, every cell lands in a non-confirmatory bucket, and no cell reaches confirmatory. The evidence here is a single two-model, five-benchmark case study, and F1--F5 is an illustrative, deliberately non-exhaustive starting taxonomy -- not a comprehensive partition of audit failures. We position the gate as a withholding and disclosure protocol for assurance-grade evidence, supplementary to not a replacement for classical construct-validity evidence, and not as a route to benchmark-validity verdicts.