Relative Value Learning

This paper proposes Relative Value Learning (RV), which estimates differences between state values directly through an antisymmetric function, Δ(s_i, s_j) = V(s_i) − V(s_j), instead of learning absolute state values independently. It defines a pairwise Bellman operator and proves that it is a γ-contraction with a unique fixed point corresponding to the true value differences. The authors derive one-step, n-step, and λ-return targets, then reconstruct generalized advantage estimation to obtain an unbiased policy-gradient estimator called R-GAE. An RV-based PPO implementation achieves competitive results against standard PPO across 49 Atari ALE games.