{"slug": "does-reasoning-improve-psychological-depth-in-large-language-models", "title": "Does Reasoning Improve Psychological Depth in Large Language Models?", "summary": "A study posted to arXiv (2611.09061v1) found no universal quality advantage from reasoning-enhanced large language models on creative writing, with human annotators slightly preferring GPT-5 over GPT-4o (60.0–62.9% across two reasoning-effort settings) while DeepSeek-R1 trailed DeepSeek-V3 at 42.9% in 60 blinded pairwise evaluations. Seven human annotators showed low agreement (Krippendorff's alpha = 0.070), which the authors attribute to genuine evaluative subjectivity, while an ensemble LLM-as-a-Judge displayed a strong pro-reasoning bias, favoring reasoning models in 98.3% of pairs versus 55.2% of human votes across its five configurations. The authors conclude that at the frontier, reasoning-related quality differences are small enough to be dominated by reader subjectivity, and that LLM-as-a-Judge evaluators can obscure that subjectivity in nuanced creative-generation tasks.", "body_md": "arXiv:2611.09061v1 Announce Type: new \nAbstract: Benchmark gains from reasoning-enhanced large language models encourage the assumption that these models should broadly perform better on complex generative tasks, but this remains underexplored in open-ended, creative domains shaped by nuances of reader experience. We tested this assumption through the lens of psychological depth in short stories. We compared DeepSeek-R1 vs. DeepSeek-V3, our closest available contrast for reasoning-oriented post-training, and GPT-5 vs. GPT-4o, through 60 blinded pairwise evaluations in which seven human annotators and an ensemble LLM-as-a-Judge rated prompt-matched story outputs. Human evaluations showed no universal reasoning advantage: GPT-5 was slightly preferred over GPT-4o (60.0-62.9% across two reasoning-effort settings), whereas DeepSeek-R1 fell behind DeepSeek-V3 (42.9%). Although this pattern was marked by low inter-annotator agreement (Krippendorff's $\\alpha = 0.070$), our annotation-quality checks suggested genuine evaluative subjectivity, plausibly reflecting distinct annotator decision schemes rather than annotation failure. In contrast, the LLM-as-a-Judge showed a strong pro-reasoning bias (98.3% of pairs vs. 55.2% of human votes), consistent across its five configurations and partly tracking sentence length. These findings suggest that, at the frontier, reasoning-related quality differences are small enough to be dominated by reader subjectivity, and that LLM-as-a-Judge evaluators can obscure this subjectivity in nuanced creative-generation tasks.", "url": "https://wpnews.pro/news/does-reasoning-improve-psychological-depth-in-large-language-models", "canonical_source": "https://arxiv.org/abs/2611.09061", "published_at": "2026-11-09 05:00:00+00:00", "updated_at": "2026-11-09 05:21:18.822854+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "artificial-intelligence"], "entities": ["DeepSeek-R1", "DeepSeek-V3", "GPT-5", "GPT-4o", "arXiv", "DeepSeek", "OpenAI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/does-reasoning-improve-psychological-depth-in-large-language-models", "markdown": "https://wpnews.pro/news/does-reasoning-improve-psychological-depth-in-large-language-models.md", "text": "https://wpnews.pro/news/does-reasoning-improve-psychological-depth-in-large-language-models.txt", "jsonld": "https://wpnews.pro/news/does-reasoning-improve-psychological-depth-in-large-language-models.jsonld"}}