Advancing AMIE towards expert-level audio-visual clinical consultations

Google Research says AMIE is no longer just a text chat box for medical questions. In its new video form, the system can hold a live consultation, watch for non-verbal clues, guide a patient through a virtual exam, and keep reasoning in real time.

That matters because so much of medicine happens off the page. A patient’s breathing, posture, discomfort, or the way they move their body can shape a diagnosis. Text strips that away. Google’s own argument is that video brings some of that clinical texture back into the loop.

To make that work, AMIE (Video) splits the job into three parts. One agent talks to the patient. One keeps updating the diagnosis and care plan in the background. A third watches the audio and video stream for signs such as distress, physical findings, and other clinically relevant cues. The company says this setup helps the system stay responsive without giving up on slower reasoning.

Google tested the system in a randomized study with 100 scenarios, 300 live consultations, 15 trained patient actors, 10 board-certified primary care physicians in the video arm, and an independent panel of 20 experienced primary care physicians doing the scoring. Across history-taking, diagnostic accuracy, management, and communication, AMIE (Video) was rated on par with the doctors. It also did better than both the text-only version and the doctors at eliciting physical signs and leading virtual exam maneuvers.

The patient actors also preferred the video setup. They said it was easier to use and better for explaining health concerns, and they rated AMIE (Video) well on empathy, rapport, and confidence in care. But Google is careful not to oversell it. These were simulated encounters, not real patients, and the company says the system still has perceptual errors and occasional technical issues. The next step, plainly, is harder: proving this holds up outside the acting school version of medicine.