AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

AgentOPSD is a critic-free method for turn-level credit assignment in long-horizon agentic reinforcement learning. It aggregates token-level teacher-student log-probability gaps into turn-level evidence, then recursively updates a Bayesian belief state in log-odds space. The resulting marginal belief revisions identify pivotal turns and reweight sparse outcome supervision without extra rollouts or an additional critic. According to the supplied abstract, experiments cover ALFWorld, WebShop, and Search-QA with Qwen2.5-3B and Qwen2.5-7B; the 7B model reaches 89.1% success on ALFWorld and outperforms GRPO and self-distillation baselines.