Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling

The paper introduces Hierarchical Landmark Sparse (HiLS) Attention, a chunk-wise sparse mechanism that learns chunk selection directly from the language-modeling loss. Each query attends independently to retrieved chunks, then combines chunk-specific outputs using retrieval scores that participate in the forward computation. According to the supplied abstract, HiLS matches or sometimes exceeds full attention at in-domain context lengths, extrapolates beyond 64 times the training context length with 90% retrieval accuracy, and can convert existing full-attention models through lightweight continued pretraining while preserving in-domain performance.