Evaluating Long-Context Question & Answer Systems

Eugene Yan just published a lengthy breakdown of why evaluating question-answering systems gets messy once documents stop being tidy paragraphs and start being novels, legal contracts, or stacks of research papers. The core problem isn't whether a model can find an answer. It's whether you can trust the answer once you have it, and whether that answer is actually useful to a human who needed it.

Yan splits evaluation into two dimensions that sound similar but pull in different directions: faithfulness and helpfulness. Faithfulness means the answer sticks strictly to what's in the source text, no outside knowledge, no invented details. This matters enormously for things like insurance forms or lease agreements, where a model needs to say "I don't have that information" rather than confidently making something up. Helpfulness is the opposite failure mode's opposite virtue — an answer can be perfectly grounded in the document and still be useless. Yan's example: asking what happens if a tenant misses a payment, and getting back "Clause 4.2 addresses missed payments." Technically faithful. Completely unhelpful.

The piece leans hard on a handful of existing benchmarks — NarrativeQA, QASPER, HELMET — to show how question design shapes what you're actually testing. NarrativeQA generates questions from summaries rather than full texts specifically so models can't just pattern-match nearby sentences; QASPER does something similar with academic abstracts. HELMET, meanwhile, tests whether models can find evidence buried in the middle of a document versus at the start or end, poking at the well-known