On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning

A paper titled "On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning" examines how on-policy reinforcement learning for large language models breaks its own assumption that training responses are sampled from the current policy, because an inference engine samples the responses while a separate trainer computes the gradients. The work attributes the resulting training-inference mismatch between the two systems to that split and traces how the mismatch evolves during training.

On-policy reinforcement learning for large language models assumes that training responses are sampled from the current policy. In practice, an inference engine samples the responses and a separate trainer computes the gradients, and the training-inference mismatch between the two makes the training