{"slug": "are-we-optimizing-the-training-process-or-the-models-ability-to-reason", "title": "Are We Optimizing the Training Process—or the Model’s Ability to Reason?", "summary": "MemoraX AI, working with Tianjin University and Alibaba, has published a NeurIPS 2026 research paper arguing that reinforcement learning for large language models should optimize inference-time behavior rather than training policy alone. The work introduces MIPI, an objective requiring each update to preserve or improve inference performance, and MIPU, a two-stage method that compares policies before and after updates and evaluates candidate responses. The team tested the approach on Qwen3-1.7B and Qwen3-4B models under FP8 training across mathematical reasoning benchmarks.", "body_md": "In large language model reinforcement learning, better training metrics do not always translate into better reasoning performance.\n\nA lower training loss, higher reward, or continuously updated policy may look like progress. But when the model is evaluated in a real inference environment, the improvement can be much smaller—or may not appear at all.\n\nWhy does this happen?\n\nOne possible explanation is the mismatch between training and inference. Differences in sampling strategies, numerical precision, inference engines, and system configurations can cause the model to behave differently during training and deployment.\n\nThis raises a fundamental question:\n\nShould we optimize the training policy itself, or should we directly optimize how the model performs during inference?\n\n**MemoraX AI, together with Tianjin University and Alibaba, explores this question in a research project for NeurIPS 2026:**\n\n**The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning**\n\nPaper:\n\n[https://arxiv.org/abs/2606.29526](https://arxiv.org/abs/2606.29526)\n\nProject page:\n\n[https://anitaleungxx.github.io/MIPU](https://anitaleungxx.github.io/MIPU)\n\nMost reinforcement learning pipelines focus on improving the training process through reward design, policy updates, sampling strategies, and optimization objectives.\n\nHowever, the training environment and the inference environment are not always identical.\n\nTraining may involve systems such as vLLM, SGLang, Megatron, or FSDP, while inference may use a different runtime or sampling configuration. Even small differences in implementation can affect the model’s output distribution.\n\nAs a result, a policy that looks better during training may not necessarily produce better answers during inference.\n\nThis training–inference mismatch can lead to several problems:\n\nThe key question is therefore not simply whether the training policy has improved.\n\nThe more important question is:\n\nDid the model actually become better at solving the task?\n\nMIPI starts from a simple idea:\n\nTraining updates should be evaluated by how they affect the model’s actual inference behavior.\n\nLet μₖ denote the inference policy after the k-th update, πₖ denote the training policy at step k, and J denote the model’s performance on the target inference task.\n\nMIPI aims to enforce:\n\n**J(μₖ₊₁) − J(μₖ) ≥ 0**\n\nIn other words, each effective update should preserve or improve the model’s inference performance.\n\nThe change in inference performance can be decomposed as:\n\n**J(μₖ₊₁) − J(μₖ)**\n\n**= [J(μₖ₊₁) − J(πₖ₊₁)]**\n\n**+ [J(πₖ₊₁) − J(πₖ)]**\n\n**+ [J(πₖ) − J(μₖ)]**\n\nThis decomposition highlights three different factors:\n\nThis perspective suggests that improving the training policy alone is not enough. We also need to understand how training updates affect the model’s actual inference behavior.\n\nTo make MIPI actionable, the paper introduces MIPU.\n\nMIPU consists of two main stages:\n\nMIPU compares the behavior of the policy before and after an update.\n\nIt considers the probability assigned to candidate responses by both the current training policy and the updated policy:\n\n**πᵏθ / μₖ = (πₖ / μₖ) × (πθ / πₖ)**\n\nThis allows MIPU to separate changes caused by the training update from differences between training-time and inference-time behavior.\n\nThe first stage produces a set of candidate models or candidate updates for further evaluation.\n\nMIPU then evaluates the actual responses generated by each candidate.\n\nAn update is more valuable when it:\n\nThe goal is not to reject as many updates as possible.\n\nRejecting too many candidates can prevent the model from exploring new reasoning paths. Accepting too many low-quality updates can introduce instability.\n\nMIPU therefore aims to balance reliable improvement with continued exploration.\n\nThe method was evaluated on Qwen3-1.7B and Qwen3-4B models under FP8 training settings.\n\nThe experiments covered several mathematical reasoning benchmarks, including:\n\nMIPU was compared with standard GRPO, MIS, and learning-rate decay.\n\n| Model | GRPO | MIS | Learning-rate decay | MIPU | Improvement over GRPO | \n|---|---|---|---|---|---|\n| Qwen3-4B | 64.42% | 63.42% | 65.66% | 66.71% | +2.29 points | \n| Qwen3-1.7B | 50.86% | 51.73% | 52.23% | 53.97% | +3.11 points | \n\nMIPU achieved higher average accuracy on both model sizes.\n\nOn Qwen3-4B, MIPU reached 66.71% average accuracy, improving over standard GRPO by 2.29 percentage points.\n\nOn Qwen3-1.7B, MIPU reached 53.97%, improving over GRPO by 3.11 percentage points.\n\nThe method also showed more stable training trajectories, suggesting that the improvement was not limited to a single evaluation point.\n\nThe paper also evaluates the contribution of each stage:\n\n| Method | Average accuracy | \n|---|---|\n| Standard GRPO | 64.42% | \n| Step 1: candidate generation only | 65.36% | \n| Step 2: candidate evaluation only | 62.81% | \n| Full MIPU | 66.71% | \n\nThe results suggest that candidate generation and candidate evaluation play complementary roles.\n\nStep 1 helps identify potentially useful updates. Step 2 helps determine whether those updates actually improve inference behavior.\n\nTogether, they allow MIPU to filter ineffective updates while preserving the model’s ability to explore new solutions.\n\nThis work focuses on 1.7B and 4B models and mathematical reasoning tasks. Several directions remain open:\n\nThe broader message is simple:\n\nTraining optimization should ultimately be measured by the quality of the model’s real-world behavior.\n\nMIPI and MIPU offer one possible direction: making the connection between training updates and inference performance more explicit, measurable, and stable.", "url": "https://wpnews.pro/news/are-we-optimizing-the-training-process-or-the-models-ability-to-reason", "canonical_source": "https://dev.to/memorax_ai/are-we-optimizing-the-training-process-or-the-models-ability-to-reason-1894", "published_at": "2026-10-27 10:03:59+00:00", "updated_at": "2026-10-27 10:13:44.844585+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["MemoraX AI", "Tianjin University", "Alibaba", "NeurIPS 2026", "Qwen3-1.7B", "Qwen3-4B", "MIPI", "MIPU"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/are-we-optimizing-the-training-process-or-the-models-ability-to-reason", "markdown": "https://wpnews.pro/news/are-we-optimizing-the-training-process-or-the-models-ability-to-reason.md", "text": "https://wpnews.pro/news/are-we-optimizing-the-training-process-or-the-models-ability-to-reason.txt", "jsonld": "https://wpnews.pro/news/are-we-optimizing-the-training-process-or-the-models-ability-to-reason.jsonld"}}