Preference Tuning LLMs with Direct Preference Optimization Methods

Hugging Face · 2 years ago · 48

Researchers fixed an implementation bug in the IPO (Identity Preference Optimization) preference-tuning method and compared three alignment techniques—DPO, IPO, and KTO—across two 7-billion parameter language models using different beta hyperparameter values. The best results varied significantly by model and algorithm, with DPO achieving the highest MT-Bench scores at beta values ranging from 0.01 to 0.6 depending on the base model. The findings demonstrate that hyperparameter tuning is critical for preference alignment without reinforcement learning, with all code and trained models made publicly available through the alignment-handbook repository.