Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Models

Vision-language models (VLMs) are frequently repurposed as reward functions for robotic reinforcement learning, where they are assumed to assign consistent progress scores to semantically equivalent instructions. Introducing ROBORMBENCH—a benchmark containing 2,390 real-robot trajectories and 21,673 verified paraphrases—the authors reveal that both open and proprietary VLMs exhibit severe paraphrase fragility, sometimes flipping the evaluation of identical executions between success and failure. Larger model scales or explicit reasoning do not reliably resolve the issue, underscoring the necessity of trajectory-grounded supervision for dependable robotic reward modeling.