A Practical Recipe for Semi-Supervised Federated ASR: Online Pseudo-Labels with Server Update Stabilization
Apple Machine Learning Research
Semi-supervised federated ASR uses clients’ unlabeled speech with server seed labels, but pseudo-label mistakes can compound across sequences and training rounds, causing divergence and widening the gap to fully supervised federated learning. On 9 of 11 tested pairs, the approach improves results by 20.8% on average in-domain and 10.0% cross-domain. The method changes SSFL by coupling a per-client online teacher (with optional global-to-online transition at round r) to server-side stabilization that continues training on labeled data between rounds, with stabilization sensitivity to data augmentation and batch size.
Why it matters
Semi-supervised federated learning (SSFL) trains models on clients’ unlabeled data using a teacher to generate pseudo-labels, with a small labeled seed dataset on the server. Automatic Speech Recognition (ASR) is particularly fragile here: pseudo-label errors compound across the output sequence and across training rounds into divergence, leaving a large gap to fully-supervised FL. We show that closing this gap turns on two coupled design axes—the teacher (which model generates the pseudo-labels) and the anchor (the server-side updates on labeled data that stabilize training). On the teacher…