Thoughts-as-Planning: Latent World Models for Chain-of-Thoughts Optimization via Reinforcement Planning

Researchers have introduced Thoughts-as-Planning, a framework that formalizes reasoning chain optimization in large language models as a sequential decision-making process over a latent semantic space. The method models the LLM as a partially observable environment and uses a latent world model to simulate how reasoning chain edits affect outputs, enabling planning via gradient descent or reinforcement learning. In experiments on language understanding and generation tasks, Thoughts-as-Planning outperformed existing reasoning chain tuning methods in efficiency, robustness, and generalization while offering interpretability through structured planning trajectories.

arXiv:2605.28842v1 Announce Type: new Abstract: The success of large language models LLMs across diverse NLP tasks has elevated the importance of reasoning chain optimization as a critical step in aligning model behavior with task objectives. Existing reasoning chain tuning methods often rely on black-box heuristics or gradient-free search, which lack interpretability, generalization, and sample efficiency. In this work, we introduce \textbf{Thoughts-as-Planning}, a novel framework that formalizes reasoning chain optimization as a sequential decision-making process over a latent semantic space. We model the LLM as a partially observable environment and learn a latent world model that simulates the effect of reasoning chain edits on downstream outputs. A proximity-preserving embedding space is constructed to encode reasoning chain-response dynamics, enabling planning via gradient descent or reinforcement learning. Our method supports multi-scale abstraction, allowing reasoning chain edits at token, segment, and instruction levels to be integrated into a unified planner. Through extensive experiments on language understanding and generation tasks, we demonstrate that Thoughts-as-Planning outperforms state-of-the-art reasoning chain tuning baselines in efficiency, robustness, and generalization, while offering interpretability through its structured planning trajectory. Our code is available at https://github.com/FastLM/Thoughts-as-Planning.