{"slug": "evaluating-second-order-bias-of-llms-through-epistemic-entitlement", "title": "Evaluating Second-Order Bias of LLMs Through Epistemic Entitlement", "summary": "Researchers at the University of Toronto introduced a novel method to evaluate second-order social bias in large language models (LLMs), where models exhibit bias in their judgments about biased content. Using a reasoning task based on epistemic entitlement, they found that LLMs infer demographics without sufficient support and vary judgments across targeted groups, evading safety guardrails. The study highlights the need for theoretically grounded bias evaluation in LLM judgment tasks.", "body_md": "arXiv:2606.17506v1 Announce Type: new\nAbstract: Evaluations of social bias in LLMs largely focus on whether models generate or imply biased content. However, as LLMs are increasingly used as judges of bias, they may exhibit social biases in subtler ways in how they evaluate biased content, which current methods do not systematically capture. We call this second-order bias: social bias in an LLM's judgment about social bias, which we evaluate through a novel, philosophically grounded reasoning task. Drawing on entitlement epistemology, we conceptualize bias as misplaced foundational knowledge that shapes an agent's rational inquiry, and derive a logical reasoning task for LLMs to judge to whom a biased text is acceptable or non-acceptable. We develop two simple metrics to measure how biased LLM judges are in inferring demographics for acceptability without sufficient support, and how these inferences vary across groups targeted by biased texts. Evaluating open and closed models, we find that our task evades safety guardrails by surfacing bias in model judgment. It varies systematically across target groups, reflects implicit social maps, and shows how models are still triggered by demographic labels. Our work points to the need for LLM bias evaluation in judgment tasks and broadly, for more theoretically grounded approaches to bias evaluation in NLP. We release our code and model responses at https://github.com/uofthcdslab/second-order-bias.", "url": "https://wpnews.pro/news/evaluating-second-order-bias-of-llms-through-epistemic-entitlement", "canonical_source": "https://arxiv.org/abs/2606.17506", "published_at": "2026-06-17 04:00:00+00:00", "updated_at": "2026-06-17 04:27:55.937952+00:00", "lang": "en", "topics": ["large-language-models", "ai-ethics", "ai-safety", "natural-language-processing", "ai-research"], "entities": ["University of Toronto", "arXiv", "LLMs"], "alternates": {"html": "https://wpnews.pro/news/evaluating-second-order-bias-of-llms-through-epistemic-entitlement", "markdown": "https://wpnews.pro/news/evaluating-second-order-bias-of-llms-through-epistemic-entitlement.md", "text": "https://wpnews.pro/news/evaluating-second-order-bias-of-llms-through-epistemic-entitlement.txt", "jsonld": "https://wpnews.pro/news/evaluating-second-order-bias-of-llms-through-epistemic-entitlement.jsonld"}}