VaultGemma: The world's most capable differentially private LLM

Google Research · 1 year ago · 40

Google and Anthropic researchers developed VaultGemma, a 1 billion parameter language model trained with differential privacy using new scaling laws that model the compute-privacy-utility trade-offs. The model was trained with sequence-level differential privacy guarantees of (ε ≤ 2.0, δ ≤ 1.1e-10) and shows no detectable memorization of training data. The weights are open-sourced on Hugging Face and Kaggle, though current privately-trained models achieve utility comparable to non-private models from approximately 5 years ago.