Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization

The paper introduces Contrastive Policy Optimization (CPO), which shapes advantages in reinforcement learning with verifiable rewards using token-level disagreement between reference-guided and vanilla generation distributions. The authors argue that this contrastive signal better separates useful uncertainty from confusion than entropy. They formulate on-policy distillation as a special case of CPO with an external teacher, and report that CPO addresses the zero-advantage problem. Experiments on in-domain and out-of-domain benchmarks are reported to outperform entropy-based RLVR methods while preserving generalization, although the supplied abstract gives no quantitative results.