When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learning

The paper identifies Positive-Credit Contamination in critic-free LLM reinforcement learning: methods such as GRPO assign the same trajectory-level advantage to every token, potentially reinforcing low-probability tokens that are contextually wrong. It proposes Tail-Aware Credit calibratiOn (TACO), which estimates token-level tail risk from local generation context and reduces positive credit for risky tokens without deleting their gradients. This preserves the possibility that useful rare patterns accumulate reinforcement while incidental noise is dampened. Experiments across three LLMs and eight benchmarks reportedly show consistent gains over GRPO-style baselines, including improved stability during long-horizon RL.