AI advice suppresses people's willingness to say "I don't know", even when the advice is wrong and accuracy is incentivized

A study of 3,132 participants found that access to AI advice, even when wrong, nearly eliminated people's willingness to say "I don't know" to difficult questions, reducing accuracy by two-thirds while nearly doubling confidence. Incentivizing accuracy reduced reliance on AI but still left participants far less willing to suspend judgment than without AI. The findings suggest AI may alter metacognitive thresholds for deciding when to answer.

arXiv:2607.13562v1 Announce Type: new Abstract: Knowing when to say "I don't know" is fundamental to human judgment, yet AI assistants offer a fluent answer to almost any question. In five experiments N = 3,132; four preregistered, one direct replication , participants answered difficult questions and could always decline to respond. We engineered the questions so that AI advice was wrong, separating AI use from its accuracy. Merely having access to AI nearly eliminated participants' willingness to suspend judgment, and this held whether the advice was actively requested or simply displayed. Consequently, participants answered more questions but were correct about a third as often as when AI was unavailable-yet their confidence nearly doubled. Incentivizing accuracy and penalizing inaccuracy led participants to seek and follow AI advice less, answer more accurately, and suspend judgment more often, though still far less than when AI was unavailable. As AI suggestions grow ubiquitous and unsolicited, they may not simply affect answer accuracy; they may even alter the metacognitive threshold at which people decide whether they know enough to answer.