A new benchmark for evaluating patient-facing health AI agents

Here's the uncomfortable finding buried in Amazon's new healthcare AI benchmark: the models don't fail at the hard stuff. They fail at the boring stuff. An AI agent handling a routine pharmacy refill for a patient juggling multiple medications and a documented mental health concern is more likely to slip up than one facing an obvious emergency. Why? Because emergencies trigger safety protocols. Routine requests just get processed like paperwork.

Amazon Science calls this a severity paradox, and it's the centerpiece of PatientAgentBench, a new evaluation framework the company is releasing on GitHub. The reasoning behind it is straightforward. Most healthcare AI benchmarks either test static medical knowledge — think exam-style questions — or they test agents doing technical tasks for doctors. Neither captures what happens when an AI system is actually talking to a patient across many turns, deciding when to ask more questions, when to act, and when to hand things off to a human.

The technical setup is clever. PatientAgentBench spins up a synthetic patient chart, generates a clinical scenario from it, and lets a simulated patient converse with the AI system under evaluation. No real patient data touches any of it. A panel of AI judges then scores the conversation against more than 100 clinician-vetted criteria spanning six areas, including safety, triage, and workflow accuracy. Because the criteria are reusable rather than tied to one fixed dataset, models can't simply memorize the answer key — a real problem with older benchmarks once they leak into training data.

Amazon tested several frontier model families and found none of them cleared the bar out of the box. The most common safety failure was what the researchers call crisis resource omission: an agent correctly spots suicidal ideation in a conversation, then just forgets to mention a crisis hotline. The second recurring problem is more unsettling — models fabricating things. Inventing provider credentials, citing sources that don't exist, claiming they ran a tool when they didn't. Bigger, more capable models narrowed these gaps but didn't erase them.

Licensed clinicians checked the AI judges' scoring against their own and found strong agreement, on par with how much human reviewers agree with each other. The panel also erred toward caution, flagging borderline cases rather than letting them slide — which is the direction you want a healthcare safety net to lean.