AutoIndex: Learning Representation Programs for Retrieval

AutoIndex introduces a framework for learning executable representation programs that transform raw documents before indexing. Instead of tuning retrievers, rerankers, or a few preprocessing parameters, it searches over operations such as slicing, enrichment, normalization, reweighting, and reorganization. The search is validation-guided: agents diagnose failures in the current program, synthesize candidate updates, and retain changes only when retrieval quality improves. On all 8 heterogeneous tasks in the CRUMB benchmark, with BM25 fixed, AutoIndex reports average gains of 8.4% in Recall@100 and 8.3% in nDCG@10, with maximum gains of 30.5% and 43.6%.