04:00
2026-09-29
machinebrief.com
ai-safety
Auditing System-1 Models on Biosecurity-Relevant Benchmarks: Calibration, Selective Prediction, and Permutation Instability in a Non-Generative Model
An audit of one commercial non-generative "System-1" model on 6,020 multiple-choice biosecurity-relevant items found pooled expected calibration error of 0.034 and pooled AUROC of 0.820 once the vendo…