A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets

The paper introduces HOLA, or Hippocampal Linear Attention, which augments a delta-rule recurrent state with a bounded exact key-value cache. The recurrent state handles linearly compressible structure, while the cache preserves associations that would otherwise be overwritten. Cache writes are selected using the committed residual magnitude, beta times the norm of e, without a learned eviction module; reads use a decoupled RMSNorm-gamma mechanism for sharper retrieval. In a 340M-parameter model trained on 15B SlimPajama tokens, WikiText perplexity falls from 27.32 to 22.92, while LAMBADA improves from 30.95 to 30.26. HOLA also reports stronger long-context needle recall through 32k tokens.