Can AI Agents Conduct Open-Ended AI Research? Early Evidence from Two Case Studies

The paper introduces “shadow evaluations”: frontier AI agents are assigned the central open-ended research question from two unpublished NeurIPS 2026 submissions, given six days and thousands of dollars in compute, and evaluated by the original authors. The agents completed all engineering without human assistance but failed to make substantial progress on either research question. The authors identify five recurring failure modes: poor judgment about the publication bar, uncreative responses to design weaknesses, ineffective backtracking, poor resource awareness, and instruction drift. A second model and scaffold reproduced the failures.