Direct Preference Optimization Beyond Chatbots
Hugging Face
Researchers applied Direct Preference Optimization (DPO) to reduce text degeneration in a specialized OCR model, using the model's own failure outputs as rejection training signals rather than discarding them as noise. DPO reduced degeneration rates across five model families by an average of 59.4%, with peak improvement of 87.6%, compared to supervised fine-tuning alone. The technique demonstrates that DPO can address specific failure modes in structured generation tasks without requiring human preference annotations, expanding its application beyond chat alignment.
Related stories
Preference Tuning LLMs with Direct Preference Optimization Methods
Hugging Face · 2 years ago ·
42
Auditing Preference Biases and Fine-Tuning Language Models with Direct Preference Optimization on Anthropic HH-RLHF Using TRL and LoRA
MarkTechPost · 2 weeks ago ·
10
Predictive Human Preference: From Model Ranking to Model Routing
Chip Huyen · 2 years ago ·
21
Auditing Preference Biases and Fine-Tuning Language Models with Direct Preference Optimization on Anthropic HH-RLHF Using TRL and LoRA
MarkTechPost · 2 weeks ago ·
10
Predictive Human Preference: From Model Ranking to Model Routing
Chip Huyen · 2 years ago ·
21