Does Reasoning Improve Psychological Depth in Large Language Models?

A study posted to arXiv (2611.09061v1) found no universal quality advantage from reasoning-enhanced large language models on creative writing, with human annotators slightly preferring GPT-5 over GPT-4o (60.0–62.9% across two reasoning-effort settings) while DeepSeek-R1 trailed DeepSeek-V3 at 42.9% in 60 blinded pairwise evaluations. Seven human annotators showed low agreement (Krippendorff's alpha = 0.070), which the authors attribute to genuine evaluative subjectivity, while an ensemble LLM-as-a-Judge displayed a strong pro-reasoning bias, favoring reasoning models in 98.3% of pairs versus 55.2% of human votes across its five configurations. The authors conclude that at the frontier, reasoning-related quality differences are small enough to be dominated by reader subjectivity, and that LLM-as-a-Judge evaluators can obscure that subjectivity in nuanced creative-generation tasks.

arXiv:2611.09061v1 Announce Type: new Abstract: Benchmark gains from reasoning-enhanced large language models encourage the assumption that these models should broadly perform better on complex generative tasks, but this remains underexplored in open-ended, creative domains shaped by nuances of reader experience. We tested this assumption through the lens of psychological depth in short stories. We compared DeepSeek-R1 vs. DeepSeek-V3, our closest available contrast for reasoning-oriented post-training, and GPT-5 vs. GPT-4o, through 60 blinded pairwise evaluations in which seven human annotators and an ensemble LLM-as-a-Judge rated prompt-matched story outputs. Human evaluations showed no universal reasoning advantage: GPT-5 was slightly preferred over GPT-4o 60.0-62.9% across two reasoning-effort settings , whereas DeepSeek-R1 fell behind DeepSeek-V3 42.9% . Although this pattern was marked by low inter-annotator agreement Krippendorff's $\alpha = 0.070$ , our annotation-quality checks suggested genuine evaluative subjectivity, plausibly reflecting distinct annotator decision schemes rather than annotation failure. In contrast, the LLM-as-a-Judge showed a strong pro-reasoning bias 98.3% of pairs vs. 55.2% of human votes , consistent across its five configurations and partly tracking sentence length. These findings suggest that, at the frontier, reasoning-related quality differences are small enough to be dominated by reader subjectivity, and that LLM-as-a-Judge evaluators can obscure this subjectivity in nuanced creative-generation tasks.