Instrument Effects in Language-Model Honesty Evaluation: An Auditable Single-System Demonstration

arXiv cs.CL · 6 hours ago

Researchers built a controlled text-adventure environment to evaluate whether language-model honesty tests actually measure the model's behavior or are artifacts of the evaluation design itself. Adding a third response option to a two-option verdict system shifted strong claims from 38 out of 40 to 7 out of 40, while clarifying the success criterion reduced false verdicts from 18 out of 59 to 0 out of 58. The findings demonstrate that evaluation instrument choices substantially alter measured model behavior, leading the authors to propose a four-check integrity protocol for future evaluations.