Fine-tuning open LLM judges to outperform GPT-5.2

Together AI · 5 months ago

Researchers fine-tuned open-source LLM judges using Direct Preference Optimization on 5,400 preference pairs from RewardBench 2, achieving higher accuracy than GPT-5.2 on human preference alignment. GPT-OSS 120B reached 62.63% accuracy compared to GPT-5.2's 61.62%, while costing 15.3 times less and running 14 times faster. This demonstrates that smaller open-source models can match or exceed closed-source judge performance when trained on preference data, enabling cost-effective and deployable evaluation systems.