SOPO: Socratic Guided Policy Optimization for Span-Level Hallucination Detection

Researchers released SOPO (Socratic Guided Policy Optimization), a multi-turn reinforcement learning framework for span-level hallucination detection in large language models, publishing training and evaluation code plus 1.7B and 4B SOPO models on GitHub. SOPO's Socratic feedback mechanism has a teacher model supply indirect hints inferred from ground truth during rollout to ease reward sparsity on hard examples, and Turn-based Advantage Scaling penalizes verbosity bias. On RAGTruth, SOPO (4B) reached state-of-the-art performance, surpassing leading proprietary models such as GPT-5 and standard RL baselines, and posted higher reported mean scores than GRPO on four out-of-domain benchmarks, with mixed evidence against CAPO.

arXiv:2610.32212v1 Announce Type: new Abstract: Large Language Models LLMs have gained widespread adoption in various natural language processing tasks, but suffer from hallucination issues where they generate unfaithful or inconsistent content. While Reinforcement Learning with Verifiable Rewards RLVR has shown promising applications in reasoning tasks, its application to span-level hallucination detection is challenged by localization ambiguity, reward sparsity on hard examples, and uncontrolled length inflation. To address these issues, we introduce Socratic Guided Policy Optimization SOPO , a multi-turn RL framework for robust hallucination detection. The key innovation of SOPO is a Socratic feedback mechanism, where a teacher model guides the reasoning trajectories during rollout by providing indirect hints inferred from the ground truth. This teacher-guided approach enables the model to generate more effective reasoning trajectories for hard examples, thus alleviating reward sparsity. Furthermore, we incorporate Turn-based Advantage Scaling to penalize verbosity bias, encouraging the model to develop efficient and autonomous reasoning capabilities. Empirical results on RAGTruth demonstrate that SOPO 4B achieves state-of-the-art performance, surpassing leading proprietary models e.g., GPT-5 and standard RL baselines, while obtaining higher reported mean scores than GRPO on four out-of-domain benchmarks, with mixed evidence against CAPO. We release our training and evaluation code alongside the 1.7B and 4B SOPO models at https://github.com/patsnap/SOPO.