Bayesian Repetition Penalty: A Principled Adjacent-Conditional Framework for Reversing Attention Collapse in Autoregressive Language Models

The paper introduces Bayesian Repetition Penalty, a decoding and output-layer repair framework for repetition loops in autoregressive language models. It compares observed token frequency with a corpus prior using an adjacent-conditional probability construction, yielding a self-normalizing ratio and an exact logit offset without approximation. The offset is accumulated through an exponential moving average into a frozen output-layer bias, allowing a model already trapped in a repetitive attractor to be repaired without changing the training pipeline. On a 1.5B-parameter model, the authors report reducing 2-gram repetition from 0.073 to near zero while preserving generation quality.