The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

This paper argues that LLM reinforcement learning has a deeper problem than ordinary training-inference off-policy mismatch. Because inference and training engines can assign different probabilities to identical trajectories, an update that improves the training-side policy may fail to improve the inference-side policy used in deployment. The authors propose Monotonic Inference Policy Improvement (MIPI) and a two-step Monotonic Inference Policy Update (MIPU) framework. MIPU generates sampler-referenced candidate updates and selectively accepts synchronized candidates using an inference-side gap proxy. Experiments across two model scales under high mismatch reportedly improve average reasoning performance and training stability.