UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma

This paper proposes Unbounded Positive Asymmetric Optimization (UP) for reinforcement learning training of large language models. It argues that importance-sampling objectives face an exploration-stability dilemma: unclipped updates can destabilize training, while conventional clipping limits useful exploration. UP uses a stop-gradient anchor and applies asymmetric treatment: positive-advantage samples receive unclipped gradients, while negative-advantage samples retain standard clipping safeguards. The abstract reports compatibility with token-level GRPO and DAPO, sequence-level GSPO, and experiments across dense, MoE, and vision-language models. Detailed benchmarks and implementation evidence require inspection of the full paper.