Challenges for human oversight in detecting AI-assisted diagnostic malfunctions A randomized controlled study of 48 physicians found that diagnostic performance declined alongside an increasingly malfunctioning AI for cardiac auscultation, even when physicians were cognitively forced to make an initial classification before seeing the AI recommendation and even when human performance exceeded AI reliability. Physicians' confidence in their own classifications declined faster than their trust in the AI, and the authors call for specification of oversight goals and processes to ensure patient safety. Abstract Increasingly, physicians must oversee AI systems to detect system malfunctions. We studied 48 physicians in a randomized controlled study using an increasingly malfunctioning AI for cardiac auscultation. Diagnostic performance declined alongside AI reliability, even when physicians were cognitively forced to make an initial classification, and when human performance exceeded AI reliability. Physicians’ confidence declined faster than their trust in AI. Results call for specification of oversight goals and processes ensuring patient safety. Main Text Consider an AI-system screening patients for cardiac conditions. For months, it has operated reliably, improving diagnostic accuracy and efficiency. Suddenly, a malfunction causes it to miss heart murmurs, producing erroneous outputs, ultimately posing risks to patient health. Human oversight is often proposed as a safeguard against such risks