LoRA training scripts of the world, unite!

Hugging Face Blog · 2 years ago

Community members combined Pivotal Tuning and the Prodigy optimizer to improve Stable Diffusion XL Dreambooth LoRA fine-tuning techniques, releasing the updated training script through the diffusers library. The method involves inserting new tokens into text encoders during the first 50% of training epochs before continuing with standard LoRA optimization. Key practices include using adaptive learning rates for different model components, custom image captions, min-SNR gamma weighting set to 5.0, and high-quality diverse training images.