MemLearner: Learning to Query Context Memory for Video World Models

MemLearner addresses long-horizon inconsistency in interactive video world models by learning how to query useful context rather than relying on fixed, rule-based frame retrieval. Query tokens connect historical context with predicted tokens, allowing the pretrained video generation model to use its visual priors for memory selection. The authors introduce long-video data containing scene occlusions and dynamic objects, with camera-pose annotations, and combine annotated rendered videos with unannotated real-world videos during training. According to the abstract, MemLearner outperforms prior video world models on scene consistency and memory, especially in occlusion-heavy and dynamic settings.