The Open Medical-LLM Leaderboard: Benchmarking Large Language Models in Healthcare

Hugging Face Blog · 2 years ago

The Open Medical-LLM Leaderboard is a standardized evaluation platform designed to assess the performance of large language models on medical question-answering tasks across multiple datasets. The benchmark includes 1,273 USMLE test questions, 6,100 Indian medical entrance exam questions, and 500 PubMedQA questions, with accuracy as the primary evaluation metric. Performance variations across models indicate that while commercial systems like GPT-4 excel broadly, specialized gaps remain—for example, Gemini Pro shows weak performance in anatomy and dermatology despite strength in other areas.