Riemannian Attention Mechanisms for Transformers: A Theoretical Framework and Architecture Design

The paper proposes replacing Euclidean dot-product attention with learned, token-specific Riemannian metrics and specifies a Fiber Bundle Transformer architecture. According to the supplied abstract, heterogeneous metrics produce non-Gram attention scores that cannot be represented as a low-dimensional QK^T factorization. Low-rank metric factors are claimed to make geodesic computation and inversion tractable. However, the work is theoretical: it neither proves that the design prevents representational rank collapse nor provides empirical validation. The supplied arXiv identifier and August 2026 publication date are future-dated and could not be treated as independently verified here.