AutoIndex: Learning Representation Programs for Retrieval
AutoIndex is a framework that learns executable transformation programs to map raw documents into optimized representations for retrieval systems. In experiments on the CRUMB benchmark, AutoIndex improved recall and nDCG over a static BM25 baseline across all 8 tasks, with average gains of +8.4% in Recall@100 and +8.3% in nDCG@10. The approach involves validation-guided program search, iteratively refining document representations to enhance retrieval quality.
Why it matters: This work demonstrates that document representation can be explicitly optimized as part of the retrieval pipeline, potentially leading to significant improvements in retrieval performance.
Full story at: arXiv Information Retrieval ↗