Fine-tuning open LLM judges to outperform GPT-5.2

There's a strange loop at the center of modern AI evaluation: we use language models to judge language models. Together AI's new research leans into that loop rather than away from it, and the results are hard to dismiss. The company took gpt-oss 120B, OpenAI's open-weight model, and ran it through Direct Preference Optimization on 5,407 human preference pairs. The outcome: 62.63% agreement with human judgments on RewardBench 2, edging out GPT-5.2's 61.62%, while costing 15 times less per token and running 14 times faster.

The logic behind why this works is simpler than it sounds. Generating a good answer is hard — a model has to juggle instructions, facts, and tone all at once. Judging two answers side by side is a narrower task, closer to classification than composition. Together's baseline tests bear this out: before any fine-tuning, Qwen3 235B already beat GPT-5.2 out of the box at 62.63% versus 61.62%, and gpt-oss 120B trailed only slightly at 57.91%. Fine-tuning pushed gpt-oss past the closed model entirely, while the same DPO recipe actually knocked Qwen3 down a notch, to 61.28%, a reminder that preference tuning isn't free lunch across every architecture.

Dig into the category breakdown and the picture gets more interesting. Every judge, tuned or not, crushes safety classification — 91% average accuracy — because these models have all been heavily post-trained to spot harmful content already. Factuality checks come in around 85%, math reasoning around 77%. Then it falls off a cliff: precise instruction-following sits at 32.5%, and judging general response