FairCoder: Probing LLM Bias in High-Stakes Decision Making via Coding Tasks

arXiv cs.CL · 18 hours ago

Researchers introduced FairCoder, a benchmark that uses coding tasks to measure bias in large language models when making decisions in hiring, education, and healthcare. The benchmark tested over 1,000 samples across multiple LLMs and revealed previously undocumented bias patterns, such as favoring applicants from high-income families in college admissions scenarios. The study proposes FairScore, a new metric for evaluating both refusal behavior and outcome diversity, to better assess fairness in LLM decision-making systems.