Where do LLMs Fall Short in CBT-Guided Affective Reasoning?

Researchers found that large language models (LLMs) score up to 96% on CBT licensing exams but fail to apply CBT effectively in dialogue, defaulting to validation and reflection. A new metric, Protocol Leverage Force (F), shows that even guided prompting only shifts behavior by about 1%, revealing a gap between theoretical knowledge and practical application in affective reasoning.

arXiv:2607.02885v1 Announce Type: new Abstract: Cognitive Behavioral Therapy CBT provides a structured framework for understanding a user's mental state by examining the interaction between cognitive and behavioral factors. However, out-of-the-box LLMs respond fluently and empathetically, yet collapse into validation & reflection, regardless of what the user actually needs. They know theoretical CBT scoring up to 96% accuracy on licensing exam questions but fail to apply it effectively. We explore this gap with a knowledge-guided framework that treats CBT dialogue as controlled affective reasoning: user narratives are decomposed into Beck's Cognitive Conceptualization structure, grounded in clinical SNOMED CT concepts validated via Natural Language Inference, and a Multiple Chain-of-Thought MCoT strategy selection between Validation & Reflection, Socratic Questioning, or Alternative Perspectives. To measure whether such guidance actually changes behavior, we introduce the Protocol Leverage Force F , a behavior-level metric that captures how far an intervention shifts a model away from its default response. Across three open-weight LLMs and 14 RealCBT-derived case studies, evaluated with human experts, valence-arousal trajectories, and linguistic entrainment, F shows that simply introducing protocol definitions via single chain-of-thought prompting fails to change LLM behavior, while MCoT on these definitions guides strategy selection better. Still, the effect stays within 1% approx. 1.2-1.3% , and all models remain biased toward Validation & Reflection. These results show CBT knowledge alone does not ensure effective application, giving the affective-computing community instrumentation to measure where LLMs fall short.