Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

The paper introduces Single-rollout Asynchronous Optimization (SAO) for asynchronous reinforcement learning of language-model agents. It replaces GRPO-style group sampling with one rollout per prompt, adds practical value-model training designs, and applies strict double-sided token-level clipping to address off-policy drift and optimization instability. The authors report stable training for 1,000 steps and consistent improvements over GRPO and variants on SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. They also describe successful use of SAO in a simulated online-learning setting and in the agentic RL pipeline for the open GLM-5.2 model, reported as 750B-A40B.