SymptomAI: Towards a conversational AI agent for everyday symptom assessment

Google Research just published results from one of the bigger real-world tests of AI-driven symptom checking yet attempted. The project, called SymptomAI, ran on Gemini Flash 2.0 and put 13,917 consenting participants through a conversation with one of five differently-tuned versions of the agent. Each version asked questions about symptoms, then produced a differential diagnosis — a ranked list of possible conditions — plus suggested next steps, before participants went off to see an actual healthcare provider and reported back two weeks later on what they were told.

To check whether any of this was worth trusting, the team brought in three board-certified clinicians who read the conversation transcripts and wrote their own diagnoses, blind to which list came from the AI and which came from a colleague. The clinicians ended up preferring SymptomAI's diagnosis list over a fellow clinician's more than half the time. When measured against the diagnosis participants later got from their own doctor, using a top-5 accuracy check, the clinicians also judged SymptomAI's list to be right more often than the lists produced by other human clinicians. Interestingly, the gap in SymptomAI's favor widened most in the cases where the clinicians themselves said they felt least confident in their own assessment.

One detail that stands out is how much the interview style mattered. Google tested five prompting strategies, ranging from agents given free rein to ask whatever follow-up questions they wanted, to agents restricted to a fixed set of standard medical-school history-taking questions, to a bare, unprompted chatbot representing what people get today when they just type symptoms into an LM. Every version that actively asked follow-up questions beat that unprompted baseline by a wide margin, which is a fairly blunt reminder that the value here isn't really the language model's medical knowledge — it's the interviewing.

The study also tried something less obvious: pairing SymptomAI's diagnoses with Fitbit data collected for up to 30 days before each conversation. For cases SymptomAI flagged as acute respiratory infections, the wearable data showed physiological shifts — in cardiovascular function, respiration, skin temperature, sleep — that peaked right around when people reported symptoms. Google frames this as a possible path toward using AI-generated diagnoses as reference labels for analyzing population-scale health data, something that's currently too expensive to do with real clinical labels alone.

None of this comes without caveats, and Google is upfront about them. The clinicians in the study were reading static transcripts and couldn't ask their own follow-up questions, so they may have missed things a live interview would have caught. Diagnosis itself is a moving target — participants reported symptoms at whatever random moment they happened to log in, not necessarily at the most diagnostically useful point in an illness's progression. And every label generated in this study, Google stresses repeatedly, is a research artifact, not a clinical diagnosis anyone should act on directly.