Seeing beyond BMI: Estimating cardiometabolic risk with smartphone imagery

Insulin resistance is the kind of problem that hides in plain sight. It shows up years before a type 2 diabetes diagnosis, quietly wearing down blood vessels, liver function and metabolism, while a standard fasting glucose test still reads normal. Doctors have long used HOMA-IR, a formula tied to fasting insulin and glucose, to catch it early, and a score above 2.9 flags trouble. But catching it at scale has been the hard part, because the best body-composition tool for the job, a DXA scan, needs specialized equipment, costs money and delivers a dose of radiation nobody wants for routine screening.

Google Research's answer is PhotoScan, a deep learning system that pulls body composition estimates out of ordinary 2D smartphone photos. Not just body fat percentage, either. PhotoScan also estimates the Android-to-Gynoid ratio, which compares fat carried in the trunk versus the hips and thighs, and the Visceral-to-Subcutaneous ratio, which separates the metabolically dangerous fat wrapped around organs from the fat just under the skin. Both ratios correlate strongly with insulin resistance, and until now getting them meant a DXA scan, full stop.

The model was built in stages. It started with pre-training on over 35,000 UK Biobank participant records, pairing MRI-derived frontal and lateral images with DXA measurements as ground truth. From there, the team fine-tuned it on a new cohort of 677 adults, called PhotoBIA, using real smartphone photos matched to DXA scans, with an automated system picking the best frames out of 360-degree videos. Then came an independent validation on 132 people from a 30-week San Francisco trial, where participants had DXA, PhotoScan, smartwatch bioelectrical impedance readings, fasting blood labs and continuous Fitbit data all collected in parallel.

The numbers hold up. On the PhotoBIA cohort, PhotoScan's body fat percentage error came in at 2.15 versus 2.91 for the smartwatch-based BIA sensor, with A/G and V/S errors of 0.107 and 0.094. The validation cohort produced comparable results: 2.13 for body fat percentage, 0.085 for both ratios. More telling is what happened when researchers tried to actually classify insulin resistance. A baseline model using just age, sex and BMI scored an AUROC of 0.692. Adding PhotoScan's body composition estimates pushed that to 0.760, edging close to the 0.773 that full DXA data achieved, and improved the reclassification index to 0.593 against DXA's 0.748. Adding BIA data to the same baseline, by contrast, moved the needle on neither metric, because BIA only ever hands over a body fat number and misses the ratios that carry more predictive weight.

Google is careful to call this a research prototype, and it clearly is one. But the gap it's targeting is real: BMI misses meaningful differences in how fat is distributed, DXA is too impractical for everyday screening, and wearable sensors only tell part of the story. The stated next step is folding in wearable data, glucose readings and blood biomarkers alongside the photo-based estimates, aiming at a broader picture of metabolic health rather than a single number on a scale.