Distance Generalization in Transformers: Why Bother with Positional Encoding?

While length generalization typically asks whether transformers can process longer sequences, distance generalization explores a subtler question: how models perform when token-to-token spans vary within a fixed context window. Using synthetic delay copy tasks with unseen recall gaps, the authors evaluate whether relative schemes like RoPE and ALiBi genuinely improve distance resolution over models without positional encoding (NoPE). The findings map the effects of distance diversity and transfer learning, reopening fundamental questions about the necessity of explicit position markers.