A new benchmark for evaluating patient-facing health AI agents
Amazon Science
Researchers released PatientAgentBench, a new evaluation framework for AI agents that interact directly with patients in healthcare settings, using synthetic patient records and LLM-based scoring across six clinical dimensions. The benchmark evaluated multiple frontier models on thousands of conversations and found that even capable models struggle with routine clinical cases involving complex patients, often failing at triage decisions and omitting crisis resources like suicide hotlines. The framework enables researchers to identify specific safety gaps in patient-facing AI systems and provides a reusable evaluation method that prevents training data contamination through dynamically generated scenarios rather than fixed datasets.
Why it matters
PatientAgentBench generates a synthetic patient health record, a realistic clinical vignette, and a patient agent that converses with the AI system under evaluation, to capture what a patient-facing agent actually has to do.
