The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context

arXiv cs.CL · 18 hours ago

A study found that large language models maintain overall accuracy when given task-irrelevant context prepended to questions, but this masks per-example instability where random character sequences can flip individual predictions either up or down. The instability affected different examples across models despite consistent aggregate-level stability. The findings highlight tail risks in language model reliability that aggregate metrics fail to capture, suggesting the need for per-example evaluation rather than relying on overall accuracy scores.