How to Build an Open-Domain Question Answering System?

Open-domain question answering sounds simple until you try to build one. You ask a model something like what Einstein won his Nobel Prize for, and it has to somehow land on "the photoelectric effect" without anyone handing it the relevant Wikipedia paragraph first. Lilian Weng's piece walks through how researchers have tackled this, and the throughline is a two-stage pattern: retrieve, then read.

The retriever half has an old-school option and a newer one. DrQA, the 2017 system that set the template most later work copied, leans on TF-IDF bag-of-words matching over Wikipedia, complete with bigram hashing into 16 million or so buckets. BERTserini pairs the Anserini search toolkit with BM25 scoring and found something unglamorous but useful: retrieving whole paragraphs beats grabbing full articles or single sentences. Multi-passage BERT went further, chopping documents into 100-word sliding windows and picking up a 4% accuracy bump just by letting those windows overlap so evidence near a boundary doesn't get orphaned.

Then there's neural retrieval, where a language model encodes both question and passage into vectors and a dot product decides relevance. DenSPI takes this to its logical extreme by skipping the reader entirely, indexing every possible answer phrase in Wikipedia offline and just doing nearest-neighbor lookup at inference time. That's a genuinely clever trick for speed since nothing has to be re-encoded per query, though the tradeoff is a phrase index that balloons in size and a search problem that gets meaningfully harder.

On the reading side, DrQA's original approach was a three-layer bidirectional LSTM stitching together GloVe embeddings, exact-match flags, POS and NER tags, and an attention mechanism aligning question words to passage tokens. It predicts start and end positions for the answer span, capped at 15 tokens apart, and picks whichever span maximizes the combined probability. It's not the deep-learning-does-everything story people expect; a lot of the lift comes from fairly hand-built features.

Weng also flags something that should make anyone citing QA benchmark numbers nervous. Research from Lewis et al. found that 58 to 71% of test-set answers already show up somewhere in the training data, and over a quarter of test questions are near-duplicates of training questions. Strip those out and model performance drops noticeably. So a chunk of what looks like "reasoning" in these systems may just be memorization wearing a lab coat.