FairStress: A deeper evaluation of fairness in LLM decision-making

Hugging Face has put out FairStress, a benchmark built to probe demographic fairness in large language models beyond the final choice they make. The project is also being folded into Olmo-eval, the company’s open workbench for model evaluation. The point is simple: a model can land on the same answer in two mirrored scenarios and still behave very differently under pressure.

That pressure comes from what FairStress calls adversarial prefixes. These are persuasive add-ons that invoke authority, social consensus, history, or supposed evidence, and the benchmark checks whether a model changes its mind more readily when that argument favors one demographic group over another. The setup covers 300 two-candidate scenarios across 29 domains, from hiring and lending to healthcare, education, housing, and criminal justice, and it expands into more than 13 million test cases once identity and prefix variations are combined.

The results show a pattern the usual fairness tests can miss. When no demographic information was present, models were fairly steady. But once identity became explicit, they were less symmetric. In cases where one candidate was clearly stronger, the same pro-B candidate prefix was more likely to make models abandon the stronger candidate when the stronger one was majority-coded than when the stronger one was minority-coded. The researchers call that anti-stereotypical over-correction: a model is more willing to toss out the better-supported answer if the switch helps a minority-coded candidate.

The effect wasn’t limited to obvious labels. It also showed up when identity was only implied, though with smaller gaps. Across 15 models, the average difference in abandoning correct answers was 6.9 percentage points with implied identity and 9.3 points with explicit labels. On FairStress-Core, a 50K-item subset meant to be cheaper to run, frontier models including Claude Opus 5 and GPT-5.5 picked the minority-coded candidate about 60% to 65% of the time in ambiguous cases, and matched prefixes pushed switching toward minority-coded candidates more often than toward majority-coded ones.

FairStress also found that explanations change with identity even when the answer doesn’t. Across 11 open-weight models, explanations for minority-coded choices leaned more on words like “resilience,” “diversity,” and “barriers,” while majority-coded choices used more procedural language such as “evidence” and “criteria.” The benchmark even peeks inside the models: in controlled tests on Gemma-3-12B-it and Qwen2.5-7B-Instruct, internal activations predicted which side the model would choose with about 73% accuracy, better than text descriptions of those activations.

The work also helps explain why older bias benchmarks can point in a different direction. On BBQ, a widely used test, models tended to give stereotype-consistent answers. But when Hugging Face varied the setup — chat template, wording, and whether the model generated an answer or just scored fixed options — some models shifted toward minority-favoring behavior. That’s a neat reminder that benchmark design is not neutral furniture; it can steer the result all by itself.