New framework for auditing machine unlearning

Machine unlearning sounds simple on paper: strip a person's data out of a trained model without rebuilding the whole thing from scratch. In practice, proving that the forgetting actually happened has been a mess. Auditors usually can't peek inside the model or touch the original training set, so they're stuck comparing outputs statistically, and the standard tool for that — two-sample testing — turns out to be quietly broken for this exact job.

The problem is structural, not just a tuning issue. Even two models retrained from scratch on identical data will produce slightly different output distributions if you change something as mundane as batch size. A rigid two-sample test can't tell that apart from an actual privacy failure, so it throws false positives at perfectly safe models. Worse, recent theoretical work shows a model can never fully erase a data point's influence unless it retraces its entire original training run step by step — meaning a strict two-sample test can always find some trace of the 'forgotten' data, guilty or not.

Google's answer, detailed in a paper headed to AISTATS 2026, ditches the binary comparison entirely. Instead of asking 'does the unlearned model differ from a gold-standard retrained model,' their Regularized f-Divergence Kernel Tests ask a relative question: is this model statistically closer to the safe, retrained version or to the original one that memorized the sensitive data? That reframing alone sidesteps the retrain-equivalence trap, and the framework backs it with a toolbox of divergence measures — chi-squared, KL, and hockey-stick divergence among them — each tuned to catch a different flavor of data shift, from smooth outliers to sharp, localized privacy leaks.

The numbers are the interesting part. On differential privacy audits, the hockey-stick divergence version caught a known flaw in the SVT3 sparse vector mechanism using just a few thousand samples, versus the millions required by the previous best tool, DP-Auditorium. On actual unlearning evaluation, the three-sample relative test finally stopped misclassifying safely retrained models as failures — something every prior two-sample approach got wrong. But it also delivered an uncomfortable verdict: of the popular unlearning techniques tested, only the random-label method actually passed. Finetuning, pruning, and Selective Synaptic Dampening all left detectable fingerprints of the data they were supposed to erase.

Google is careful to note these were simplified implementations, not a definitive ranking of production-grade unlearning algorithms. Still, the takeaway lands: a lot of the unlearning methods currently treated as compliance-ready may not survive a properly built audit.