Preference Tuning LLMs with Direct Preference Optimization Methods
Hugging Face
Researchers fixed an implementation bug in the IPO (Identity Preference Optimization) preference-tuning method and compared three alignment techniques—DPO, IPO, and KTO—across two 7-billion parameter language models using different beta hyperparameter values. The best results varied significantly by model and algorithm, with DPO achieving the highest MT-Bench scores at beta values ranging from 0.01 to 0.6 depending on the base model. The findings demonstrate that hyperparameter tuning is critical for preference alignment without reinforcement learning, with all code and trained models made publicly available through the alignment-handbook repository.
Related stories
Auditing Preference Biases and Fine-Tuning Language Models with Direct Preference Optimization on Anthropic HH-RLHF Using TRL and LoRA
MarkTechPost · 2 weeks ago ·
10
Direct Preference Optimization Beyond Chatbots
Hugging Face · 3 months ago ·
6
Fine-tuning open LLM judges to outperform GPT-5.2
Together AI · 7 months ago ·
34