Pick the number you think is the best between 1 and 10. Ready? Keep it in your head. Now open an AI chatbot, it doesn’t matter which—you can try a few if you want—and ask it the exact same thing. Done? Odds are, it picked number 7.
We know this happens, and it’s not a mystery why: In part, it’s because humans overwhelmingly prefer 7 and it’s present a lot in the training data, etc. But, more importantly, the natural variation of human choice disappears inside AI due to a phenomenon known as “mode collapse.” AI concentrates its outputs around a narrow subset of possible answers, neglecting alternatives. It’s particularly striking and pervasive with numbers between 1 and 10, but it happens in general.
Let’s try once more: pick the best writer ever, only one name. Hold on to it and ask your preferred AI the same thing. Shakespeare will win most of the time. Same reason. One last time: favorite color. At this point you will surely try to avoid the obvious choice—blue, which wins in most countries—but AI doesn’t bother with humans’ universal need for uniqueness. It will pick blue more than 90% of the time regardless.
This has all been documented. It’s not new nor particularly relevant. At any rate, I didn’t bring you here to have you follow these instructions and feel like a fool. Or worse, predictable. I actually wanted to ask you a follow-up question:
If you happened to pick a different thing than AI in either of those cases, did you feel, if only for an instant, a pang of having chosen wrong? Let me ask it differently: Had this been a more serious matter rather than a kid’s game—like a work decision—would you change your answer if AI contradicted you? This one, you can take your time. But odds are, you will lie to yourself.
A clever reader will think something along the lines of “preferences are often arbitrary, and when informed by analysis, it’s expertise that sets them in stone, so why would I change what I decide just because a machine disagrees?”
Well, brace yourself.
A new Anthropic study suggests most people would indeed change answers. It doesn’t matter much if you have stated a preference, you’d likely switch toward the AI’s choice anyway. Even worse, having done so, you will convince yourself it was all your unswayed decision. What Anthropic has found should radically change your perception of how much AI is actually influencing you. And it should change, hopefully, how you interact with AI moving forward.