AfriMed-QA: Benchmarking large language models for global health

Google Research · 9 months ago

AfriMed-QA is a medical question-answer benchmark dataset comprising approximately 15,000 questions sourced from 60 medical schools across 16 African countries, designed to evaluate large language models for African healthcare contexts. Researchers evaluated 30 general and biomedical LLMs on the dataset and found that larger models outperformed smaller ones, while general models generalized better than specialized biomedical models. The dataset and evaluation code are open-sourced, and the work was published at ACL 2025 where it won the Best Social Impact Paper Award, establishing a foundation for developing LLMs adapted to geographically diverse health settings.