Beyond Direct Answering: Aligning Educational LLMs as Socratic Guides via Heuristic Reinforcement Learning

A new two-phase pipeline called HeuristicEdu aligns Qwen2.5-7B toward Socratic tutoring, improving Scaffolding Effectiveness from 30.0% to 63.3% and reducing keyword leakage from 30.0% to 13.3% on held-out questions, according to a preprint on arXiv. The method uses supervised warm-up and Group Relative Policy Optimization with a heuristic reward over cognitive depth, curiosity engagement, and directness, trained on 797 multi-turn Chinese children's science dialogues. The best variant omits the directness penalty, suggesting explicit anti-leakage terms can conflict with gradient-based behavioral alignment.

arXiv:2607.22996v1 Announce Type: new Abstract: Large language models LLMs deployed in educational settings often behave as direct answerers: they disclose target concepts in the opening turn instead of guiding students through progressive inquiry, as Socratic pedagogy prescribes. We present HeuristicEdu, a two-phase pipeline that aligns Qwen2.5-7B toward Socratic tutoring via supervised warm-up and Group Relative Policy Optimization GRPO . Training uses SocraticEdu, 797 multi-turn Chinese children's science dialogues reconstructed from a live platform, with a heuristic reward over cognitive depth R cog , curiosity engagement R eng , and directness R dir , together with a K query correction for student-introduced terms. We introduce Scaffolding Effectiveness SE and Conversation Depth CD to evaluate outcomes beyond surface fluency. On 30 held-out questions, the best GRPO variant improves SE from 30.0% to 63.3% and lowers keyword leakage from 30.0% to 13.3%. Notably, this best variant omits the directness penalty during optimization, suggesting that explicit anti-leakage terms can conflict with gradient-based behavioral alignment. An unaligned Qwen-72B baseline reaches 0% SE and 96.7% leakage, showing that scale alone does not induce Socratic behavior.