In large language model reinforcement learning, better training metrics do not always translate into better reasoning performance.
A lower training loss, higher reward, or continuously updated policy may look like progress. But when the model is evaluated in a real inference environment, the improvement can be much smaller—or may not appear at all.
Why does this happen?
One possible explanation is the mismatch between training and inference. Differences in sampling strategies, numerical precision, inference engines, and system configurations can cause the model to behave differently during training and deployment.
This raises a fundamental question:
Should we optimize the training policy itself, or should we directly optimize how the model performs during inference?
MemoraX AI, together with Tianjin University and Alibaba, explores this question in a research project for NeurIPS 2026:
The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning
Paper:
https://arxiv.org/abs/2606.29526 Project page:
https://anitaleungxx.github.io/MIPU Most reinforcement learning pipelines focus on improving the training process through reward design, policy updates, sampling strategies, and optimization objectives.
However, the training environment and the inference environment are not always identical.
Training may involve systems such as vLLM, SGLang, Megatron, or FSDP, while inference may use a different runtime or sampling configuration. Even small differences in implementation can affect the model’s output distribution.
As a result, a policy that looks better during training may not necessarily produce better answers during inference.
This training–inference mismatch can lead to several problems:
The key question is therefore not simply whether the training policy has improved.
The more important question is:
Did the model actually become better at solving the task?
MIPI starts from a simple idea:
Training updates should be evaluated by how they affect the model’s actual inference behavior.
Let μₖ denote the inference policy after the k-th update, πₖ denote the training policy at step k, and J denote the model’s performance on the target inference task. MIPI aims to enforce:
J(μₖ₊₁) − J(μₖ) ≥ 0 In other words, each effective update should preserve or improve the model’s inference performance.
The change in inference performance can be decomposed as:
**J(μₖ₊₁) − J(μₖ)**
**= [J(μₖ₊₁) − J(πₖ₊₁)]**
**+ [J(πₖ₊₁) − J(πₖ)]**
**+ [J(πₖ) − J(μₖ)]**
This decomposition highlights three different factors:
This perspective suggests that improving the training policy alone is not enough. We also need to understand how training updates affect the model’s actual inference behavior.
To make MIPI actionable, the paper introduces MIPU.
MIPU consists of two main stages:
MIPU compares the behavior of the policy before and after an update.
It considers the probability assigned to candidate responses by both the current training policy and the updated policy:
πᵏθ / μₖ = (πₖ / μₖ) × (πθ / πₖ) This allows MIPU to separate changes caused by the training update from differences between training-time and inference-time behavior.
The first stage produces a set of candidate models or candidate updates for further evaluation.
MIPU then evaluates the actual responses generated by each candidate.
An update is more valuable when it:
The goal is not to reject as many updates as possible.
Rejecting too many candidates can prevent the model from exploring new reasoning paths. Accepting too many low-quality updates can introduce instability.
MIPU therefore aims to balance reliable improvement with continued exploration.
The method was evaluated on Qwen3-1.7B and Qwen3-4B models under FP8 training settings.
The experiments covered several mathematical reasoning benchmarks, including:
MIPU was compared with standard GRPO, MIS, and learning-rate decay.
| Model | GRPO | MIS | Learning-rate decay | MIPU | Improvement over GRPO |
|---|---|---|---|---|---|
| Qwen3-4B | 64.42% | 63.42% | 65.66% | 66.71% | +2.29 points |
| Qwen3-1.7B | 50.86% | 51.73% | 52.23% | 53.97% | +3.11 points |
MIPU achieved higher average accuracy on both model sizes.
On Qwen3-4B, MIPU reached 66.71% average accuracy, improving over standard GRPO by 2.29 percentage points.
On Qwen3-1.7B, MIPU reached 53.97%, improving over GRPO by 3.11 percentage points.
The method also showed more stable training trajectories, suggesting that the improvement was not limited to a single evaluation point.
The paper also evaluates the contribution of each stage:
| Method | Average accuracy |
|---|---|
| Standard GRPO | 64.42% |
| Step 1: candidate generation only | 65.36% |
| Step 2: candidate evaluation only | 62.81% |
| Full MIPU | 66.71% |
The results suggest that candidate generation and candidate evaluation play complementary roles.
Step 1 helps identify potentially useful updates. Step 2 helps determine whether those updates actually improve inference behavior.
Together, they allow MIPU to filter ineffective updates while preserving the model’s ability to explore new solutions.
This work focuses on 1.7B and 4B models and mathematical reasoning tasks. Several directions remain open:
The broader message is simple:
Training optimization should ultimately be measured by the quality of the model’s real-world behavior.
MIPI and MIPU offer one possible direction: making the connection between training updates and inference performance more explicit, measurable, and stable.