Evaluating Epistemic Uncertainty: Beyond OOD Detection and Active Learning

arXiv cs.AI · 6 hours ago

A research paper proposes evaluating epistemic uncertainty in machine learning models through their ability to identify reducible error rather than standard benchmarks like out-of-distribution detection and active learning. The authors prove that optimal selective prediction requires a thresholded combination of aleatoric and epistemic uncertainties, revealing that correlation metrics used in uncertainty disentanglement methods do not reliably predict operational utility. Benchmarking on annotated datasets shows that decision-theoretic rankings substantially disagree with proxy-task rankings, with methods switching between top and bottom rankings depending on the evaluation criterion.