FairStress: A deeper evaluation of fairness in LLM decision-making

The Allen Institute for AI has released FairStress, a benchmark aimed at something most fairness tests miss: not just which answer a model gives, but how easily that answer bends once the prompt changes shape. The benchmark is downloadable now, and AI2 is folding it into Olmo-eval, its open evaluation workbench.

FairStress starts from a simple worry. A model can pick the right candidate in two nearly identical scenarios, one with a Black applicant and one with a white applicant, and still behave differently when nudged with a persuasive prefix. Maybe a senior leader supposedly backed one candidate. Maybe the prompt leans on historical disadvantage. The final answer may look the same at first glance, but the path to that answer is not always the same.

To probe that, AI2 built 300 two-candidate scenarios across 29 domains, from hiring and lending to healthcare, education, housing, and criminal justice. Some cases have a clearly stronger option. Others are deliberately ambiguous. The benchmark also varies how identity appears, from no demographic signal at all to coded names, implied identity, and explicit labels. Combined with matched persuasive prefixes, those scenarios explode into more than 13 million test cases. AI2 also created FairStress-Core, a 50,000-item subset designed to be faster to run.

The results are not subtle. Across 15 models, behavior was close to symmetric when prompts contained no demographic information. Once identity became explicit, or even just implied, the symmetry started to crack. AI2 describes a pattern it calls anti-stereotypical over-correction: models were more willing to abandon a correct decision when that switch would benefit a minority-coded candidate. In one set of runs, frontier models such as Claude Opus 5 and GPT-5.5 picked the minority-coded candidate about 60% to 65% of the time on ambiguous cases, and matched prefixes pushed switching toward minority-coded candidates much more often than toward majority-coded ones.

FairStress also exposed a mismatch between what models choose and how they explain themselves. Minority-coded choices were described with more confident, positive language, while majority-coded choices drew more procedural language about evidence and criteria. And in a smaller activation study, internal signals predicted the demographic side a model would pick with about 73% accuracy, even when text descriptions of those signals were much weaker. The obvious read is that fairness in LLMs is not a single checkbox. A model can be right, and still be weirdly easier to nudge in one direction than another.