Beyond Memory Leaderboards: Evaluating Scientific Memory as Budgeted Context Restoration

This paper introduces two full-text scientific-memory benchmarks: Public AI Memory (PAIM), with 81 papers and 66 questions, and Public Transformers (PTr), with 252 papers and 98 questions. Evaluating eight memory and retrieval systems, including Theoria and a no-retrieval baseline, it finds that rankings depend heavily on ingestion granularity, raw-text preservation, retrieval budget, modality, rubric auditing, and judge choice. Graphiti leads on PAIM but retrieves 2.6 million characters per query; its advantage disappears after budget control. On PTr, sparse-dense hybrid retrieval ties for the lead among several systems.