AfriMed-QA: Benchmarking large language models for global health
Google Research
Google Research built AfriMed-QA, a huge medical Q&A dataset from 60 African medical schools across 16 countries. It's meant to test whether AI health tools actually work outside Western contexts, and the results are messy.
Based on reporting by Google Research — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Most medical AI benchmarks come from the same places: US medical boards, Western clinical notes, English exam questions written for a specific kind of student. Google's AfriMed-QA project pokes a hole in that comfort zone by asking a blunt question: do these models actually hold up when the diseases, drug names, and cultural context change?
The dataset itself is substantial. Roughly 15,000 questions pulled from 621 contributors across more than 60 medical schools in 12 African countries, spanning 32 specialties from neurosurgery to infectious disease. It mixes multiple-choice exam questions, long-form short-answer items, and everyday consumer health queries — the kind of thing a patient might actually type into a search bar. Google built on a crowd-sourcing platform originally designed by Intron Health for collecting accented African speech data, which is a nice bit of infrastructure reuse.
The results complicate some assumptions. Bigger models beat smaller ones, which isn't shocking, but it's bad news for anyone hoping to run lightweight, on-device models in low-resource clinics where bandwidth and hardware are constrained. More interesting: general-purpose LLMs outperformed models specifically fine-tuned on biomedical data. Google's researchers suspect the specialized models may have overfit to the narrow slice of medical text they were trained on, making them brittle outside that lane — a caution against assuming
My take — AI-written commentary, not fact-checked reporting
I'll say the quiet part: a benchmark that admits over half its exam questions come from one country (Nigeria) and calls itself pan-African is honest, not disqualifying — most 'global' AI benchmarks don't even bother with that disclosure. The real story here isn't the leaderboard, it's that specialized biomedical models got beaten by general ones, which should worry anyone betting big on narrow fine-tuning as the future of medical AI. Open-sourcing this dataset is the right call, and I'd rather see ten imperfect regional benchmarks than one more polished Western-only exam.
Read more about this at: Google Research