Enhancing Language Model Reasoning with Structured Multi-Level Modeling

A new arXiv paper (2611.00616v1) proposes Multi-Level Reasoning (MLR), a method that splits long chain-of-thought generation into a high-level planner that emits structured step descriptors and a low-level executor that reasons conditioned on them, using a lightweight LoRA module for the planner on top of the base model. The authors train MLR with iterative Step-DPO, a process-level preference optimization scheme using Twisted Sequential Monte Carlo (TSMC) for stepwise supervision, and report that under a reduced data budget of 10% SFT and 5% preference data relative to the DeepSeek-R1 distillation setup, MLR outperforms SFT-based distillation and strong RL/preference-optimization baselines across multiple base models on math, science and logical reasoning benchmarks. MLR also degrades more slowly on long-horizon reasoning, which the authors attribute to stronger robustness under extended CoT generation.

arXiv:2611.00616v1 Announce Type: new Abstract: Inference-time scaling enhances a model's reasoning by extending its chain-of-thought CoT . However, existing approaches typically rely on a single policy trained with outcome-reward reinforcement learning RL , which often suffers from long-horizon plan failures where the implicit plan drifts from valid strategies, especially for small LMs with limited capacity. To address this, we propose Multi-Level Reasoning MLR , which reformulates long-CoT generation as a two-level stochastic process. A high-level planner generates structured step descriptors specifying both the reasoning mode and the semantic subgoal. The low-level executor then produces detailed reasoning conditioned on these descriptors, forming an alternating plan--execute loop. To maintain scalability, we adopt a minimal design where the base model serves as the low-level policy and a lightweight LoRA module implements the high-level policy. For training, we observe that outcome-reward RL provides sparse and delayed feedback for long trajectories e.g., several thousand tokens , hindering credit assignment. We therefore introduce iterative Step-DPO, a process-level preference optimization scheme that leverages Twisted Sequential Monte Carlo TSMC to provide scalable stepwise supervision. This yields more effective training, improved stability, and higher accuracy. Extensive experiments on challenging math, science, and logical reasoning benchmarks show that, under the same reduced data budget 10% SFT and 5% preference relative to the DeepSeek-R1 distillation setup , MLR outperforms both SFT-based distillation and strong RL/preference-optimization baselines across multiple base models and tasks. Moreover, MLR exhibits slower performance degradation on long-horizon reasoning, demonstrating stronger robustness under extended CoT generation.