Direct Preference Optimization Beyond Chatbots

Hugging Face Blog · 1 month ago

Researchers applied Direct Preference Optimization (DPO) to reduce text degeneration in a specialized OCR model, using the model's own failure outputs as rejection training signals rather than discarding them as noise. DPO reduced degeneration rates across five model families by an average of 59.4%, with peak improvement of 87.6%, compared to supervised fine-tuning alone. The technique demonstrates that DPO can address specific failure modes in structured generation tasks without requiring human preference annotations, expanding its application beyond chat alignment.