When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings
AI Summary
This preprint identifies a numerical failure in ALiBi positional encoding: linearly increasing distance biases can underflow floating-point precision, forcing many attention weights to zero and making affected heads partially blind. Using pretraining experiments with 148M-parameter decoder models, the authors separate this effect from ordinary out-of-context degradation. They report substantial harm to passkey-style token retrieval but only minor changes on standard decoder benchmarks. Four training-time mitigation strategies are evaluated, with log-scaled distances providing the most consistent passkey-retrieval improvements, although default ALiBi slopes remain a strong needle-in-a-haystack baseline.
Why it's worth reading
ALiBi remains relevant to long-context model design, and this paper exposes a retrieval-critical numerical failure that standard decoder benchmarks may largely miss while offering actionable training-time mitigations.
Deep Read
1. What happened
Original fact: The paper reports a previously overlooked numerical failure in ALiBi positional encoding. Its linearly increasing distance bias can drive values beyond useful floating-point resolution, causing many attention weights to underflow to zero and leaving affected attention heads able to access only part of the context.
2. Core technology
Original fact: ALiBi adds a token-distance-dependent linear bias directly to attention scores. The paper studies finite-precision underflow after these biased scores pass through softmax and evaluates four training-time mitigations. The supplied abstract names only one of them: replacing linear distance scaling with log-scaled distances.
3. Key evidence and numbers
Original fact: The authors report comprehensive pretraining experiments with 148M-parameter decoder models and state that the failure also occurs in state-of-the-art pretrained models using ALiBi. They find substantial degradation in token or passkey retrieval but only minor effects on standard decoder benchmarks. Log-scaled distances deliver the most consistent passkey-retrieval improvement. Exact scores, context lengths, precision formats, and proportions of zeroed weights are absent from the abstract.
4. Why it matters
Analysis: If standard language-model benchmarks are largely insensitive to this failure, a model can appear healthy while losing exact long-range retrieval capability. Long-context evaluation therefore needs controlled retrieval tasks alongside semantic benchmarks, plus numerical diagnostics that distinguish intended sparse attention from accidental underflow.
5. Practical impact
Analysis: Teams training or serving ALiBi models should record arithmetic precision, inspect post-softmax zero-weight rates by layer and head, and add passkey and needle-in-a-haystack tests. Log-scaled distances are a concrete candidate for new training runs, but the abstract also says default ALiBi slopes remain a strong baseline, so replacement is not automatically justified.
6. Limitations and uncertainty
Original fact: This is presented as an arXiv preprint, and the abstract does not identify all four mitigations or provide full quantitative results. Unverified inference: Severity likely depends on floating-point format, softmax implementation, context length, and slope selection, but those dependencies must be checked against the full paper. The supplied publication date is 2026-08-04 and should also be independently verified.
7. Original sources
- arXiv abstract page: 2608.03994
- This enrichment uses only the supplied title, abstract, and metadata; no additional experimental results or citations have been inferred.