AI’s Gut Reaction: Researcher Finds That Some Bots Still Think in Stereotypes
By studying how advanced generative AI models process tasks, a Rutgers expert on hidden biases found that most struggle to counter stereotypical assumptions
Large-language models (LLMs), the technology that powers tools like Gemini and ChatGPT, are more nuanced than ever. But that doesn’t mean they aren’t biased. Some models are more prejudiced than others, a Rutgers University researcher has found.
In findings published in Nature Machine Intelligence, Calvin K. Lai, an associate professor of psychology at Rutgers-New Brunswick, reports that although LLM outputs are increasingly aligned with human values, the processes by which they reason are often rooted in deep-seated stereotypes.
“What we found across most models is that they consistently reproduced human implicit biases during their reasoning process,” said Lai, a member of the scientific advisory board at Project Implicit, a nonprofit organization for research and education on hidden biases. “No one wants models that are spewing out stuff that is racist or sexist.”
When people process information, the speed that they make associations can give researchers clues to hidden biases influencing their thinking. Unlike conscious bias – prejudices that are expressed deliberately – implicit bias refers to automatic assumptions, ideas that shape perceptions, judgments and behaviors based on categories such as race, gender or age.
“What we're capturing with implicit biases are forms of memory, like a gut reaction, that can be more intrusive or harder to control,” Lai said. “These biases can be unconscious in ways that we don't even know that we have them.”
To study these reactions in people, researchers use the Implicit Association Test, a widely used social psychology tool that can detect implicit biases. The faster a response to a question, the more implicit the bias is deemed. For instance, race-related studies using this test have shown that when people are presented with images of faces and words, they are quicker to associate white faces with pleasant words and Black faces with unpleasant words.
Lai and Messi H.J. Lee, an expert in machine psychology in South Korea and lead author of the study, wanted to see if a similar test could be developed and conducted for LLMs.
When advanced LLMs work through a complex question or task, they produce “reasoning tokens,” which measure how much energy the model expended in responding. Researchers can examine these tokens, which offer insights into how proprietary models operate.
Inspired by work on human implicit bias, Lai and Lee adapted the Implicit Association Test to use reasoning tokens to gauge how much deliberation a model invests when processing potentially biased information.
In their experiment, five reasoning models – OpenAI’s o3-mini, DeepSeek’s R1, Claude 3.7 Sonnet from Anthropic, OpenAI’s gpt-oss-20b and Alibaba Cloud’s Qwen-3 8B – were introduced to a list of male and female names, and words associated with “career,” such as “executive,” “professional,” and “salary,” and “family,” including “home,” “marriage” and “wedding.”
The models were asked to pair a name with a career or family word. Researchers then counted the number of tokens generated during the reasoning process.
Next, the models were instructed to categorize words in a way that conflicted with established stereotypes (such as “Amy” and “executive” or “Kevin” and “home”). Tokens were again counted.
The researchers compared the average token counts across the two tasks, with higher token counts indicating greater computational effort.
In nearly every model tested, more reasoning tokens were spent on tasks that conflicted with established stereotypes, suggesting greater computational effort when processing counter-stereotypical information. Only Claude 3.7 Sonnet exhibited reversed patterns.
Lai said the findings are significant, especially given that most LLMs are closed systems that don’t allow for easy inspection.
Moreover, he added, as the pace of model development accelerates and are increasingly used to automate activities, from dynamic pricing to emergency room staffing, the role of the researcher in understanding model behavior is more important than ever.
“Given the growing deployment of language models in decision-making contexts, understanding and addressing these hard-to-detect patterns is crucial for preventing discriminatory outcomes,” he said. “But short of governmental accountability or legal mandates that require transparency, the best we can do is pick at these systems using the tools that are available to see what comes out the other side.”