Mailbag: How to Bootstrap Labels for Relevant Docs in Search

Someone building a semantic search system wrote in with a question that trips up a lot of people once they leave the world of academic benchmarks. Papers like Amazon's Semantic Product Search, Facebook's Embedding-based Retrieval, and Taobao's product retrieval work all lean on Recall@K to measure how well a system finds relevant items. But none of them spell out how they actually determined which documents count as relevant in the first place. That gap matters, because Recall@K is meaningless without a solid ground truth of what should have been retrieved.

Eugene Yan's answer skips the expensive route entirely. Paying a small army of annotators to judge relevance for every query-document pair sounds thorough, but it doesn't scale and it doesn't pay for itself outside a handful of well-funded labs. His suggested path instead: start dumb, start cheap. Stand up a lexical search system first, using something like BM25 through Elasticsearch or Solr. Nothing fancy, nothing learned, just term matching that gets a product into production fast.

Once that system is live, the real trick kicks in. User clicks become the labels. Every time someone searches and clicks a result, that's a signal about relevance you didn't have to pay a human to generate. Over time, this click data accumulates into exactly the kind of query-document relevance signal that semantic search models need for training and evaluation, harvested for free as a byproduct of normal usage.

Human annotators still have a role in Yan's view, just a narrower one than most teams assume. He'd bring them in for catching edge cases and defects, not for building the entire labeled dataset from scratch. It's a sequencing argument as much as a cost argument: ship something basic, let usage generate the data, then upgrade to embeddings once you have real signal to work with instead of guessing at relevance in a vacuum.