PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents
AI Summary
PAST-Bench evaluates whether persistent personal agents convert retained cross-session experience into better future behavior. It contains 26 scenarios and 204 episodes spanning memory, procedural reuse, information gathering, and updates, and compares matched runs with experience retention enabled or disabled. Experiments cover seven base models and four agent frameworks while checking both downstream gains and evidence for the intended save-retrieve-update pathway. The authors also introduce Hermes+, adding five targeted interventions to Hermes. According to the abstract, Hermes+ improves average gains, particularly when outdated state must be replaced, although results remain capability- and model-dependent.
Why it's worth reading
Persistent memory is becoming standard in personal agents, and this benchmark distinguishes apparent performance gains from improvements actually supported by the intended save, retrieve, and update process.
Deep Read
1. What happened
Original facts: The authors introduce PAST-Bench, which compares matched runs with retained experience enabled or disabled across ordered, fresh-session tasks. It includes 26 scenarios and 204 episodes covering memory, procedural reuse, information gathering, and updates. The reported evaluation spans seven base models and four agent frameworks. The authors also develop Hermes+, an extension of Hermes.
2. Core technology
Original facts: The benchmark measures both later-task gains and evidence that those gains follow the intended save, retrieve, and update pathway. Hermes+ adds five targeted interventions at different stages of the agent loop, although the supplied abstract does not describe their implementation.
Analysis: Separating outcome scores from pathway evidence can reveal whether an agent truly uses persistent experience or merely performs better because of unrelated run-level variation.
3. Key evidence and numbers
Original facts: PAST-Bench contains 26 scenarios and 204 episodes and evaluates seven base models with four agent frameworks. The abstract reports real but uneven improvement from retained experience. Agents with similar headline gains can show substantially different pathway evidence. Hermes+ uses five interventions and reportedly improves most strongly on tasks requiring outdated state to be replaced.
Missing information: The abstract provides no model or framework list, absolute scores, average effect sizes, variance, significance tests, costs, or intervention ablations. Hermes+'s advantage therefore cannot be quantified from the supplied material.
4. Why it matters
Analysis: A long-running agent must do more than accumulate records: it must retrieve relevant experience, apply it correctly, and overwrite stale state when circumstances change. PAST-Bench's process diagnostics could expose failures hidden by aggregate task scores and provide a narrow, testable operationalization of recursive self-improvement.
5. Practical impact
Analysis: Agent builders could adapt the retention-on versus retention-off methodology to evaluate memory stores, skill libraries, and preference systems, then localize failures to saving, retrieval, or updating. Reliable replacement of stale information is especially important for assistants maintaining user state because indiscriminate memory accumulation can preserve conflicting facts and routines.
6. Limitations and uncertainty
Original facts: The authors state that improvements remain dependent on capability and base model.
Analysis: Whether 204 episodes support broad cross-model and cross-framework conclusions depends on task distribution and statistical methodology. Whether pathway evidence establishes causality also depends on annotation criteria and intervention design.
Unverified: The supplied publication date is August 4, 2026, which is later than the currently verifiable context. The paper metadata, reported effects, and repository state were not independently checked, so claims in the abstract should not be treated as reproduced findings.
7. Original sources
- arXiv abstract: https://arxiv.org/abs/2608.04003
- Code repository: https://github.com/Gen-Verse/PAST-Bench
- This assessment uses only the user-provided title, abstract, date, and links; it adds no unverified authors, measurements, or citations.