Allen Institute for AI (AI2) released FairStress, a benchmark for evaluating symmetric demographic fairness in LLM decisions under adversarial demographic challenges
Benchmark result Provisional 86% confidence first seen
AI2 introduced FairStress, a new benchmark to evaluate how large language models’ decisions change under demographic identity variations and persuasive adversarial arguments. The benchmark starts with 300 two-candidate scenarios across multiple domains and expands to more than 13 million test cases by varying demographic presentation and matched adversarial prompts, and it is being integrated into Olmo-eval.