Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs

This paper introduces reinforcement learning with metacognitive feedback (RLMF) and metacognitive data selection. Both use a model’s judgments about its own performance to refine preference-optimization rankings or select valuable training examples. The authors apply the approach to faithful calibration, first calibrating self-reported confidence and then editing outputs to express uncertainty in natural, context-adaptable language. The abstract reports state-of-the-art generalizable calibration across diverse tasks, preserved accuracy, and improvements of up to 63% over standard reinforcement learning, while improving models’ assessment and expression of capability limits.