Mailbag: How to Bootstrap Labels for Relevant Docs in Search

Eugene Yan · 5 years ago · 10

An engineer asks how companies determine the total number of relevant documents when evaluating semantic search systems using Recall@K metrics, since manually annotating large datasets is expensive. The response suggests starting with lexical matching in production, then bootstrapping labels from user click data rather than hiring annotators, with human annotation reserved for edge cases and defects. This practical approach avoids the cost of large-scale annotation while building evaluation data from real user behavior.