Mamba with Hierarchical Memory: Solving the Representation Bottleneck in Long Sequence Modeling

The paper introduces Hierarchical Memory Mamba (HMM), which augments a pretrained Mamba backbone with a lightweight working memory. It extracts paragraph-level semantics from the backbone’s hidden states and compresses them into persistent long-term memory for task-relevant retrieval. On Passkey Retrieval and LongBench-E, the abstract reports 34.3–37.1% higher retrieval success and 1.6–14.2% higher reasoning accuracy than strong Mamba-based baselines, while adding only about 2% more parameters and requiring minimal training overhead.