When Memory Lies: An Empirical Study of Spatial Memory Staleness in VLM Agents

This paper studies how vision-language model agents handle conflicts between stale spatial memories and new observations in a dynamic FrozenLake testbed. It evaluates three closed-source models and three open-weight VLMs, with 1,800 staleness-detection runs and 12,000 text-mode navigation episodes. Text performance does not reliably transfer to visual grounding: reported vision F1 ranges from 0.887 to 0.067 on identical grids. In the primary GPT-4o setting, trusting raw memory causes more than twice as many deaths as using no memory. Read-time auditing reduces much of the text-mode safety cost, but unreliable visual auditing remains a major limitation.