A Decision-Only Judge Matches GPT-6 on Routine Evals for 0.36% of the Fee
StackSweep
A Carnegie Mellon team found TypeSafe’s decision-only judge matches GPT-6 on routine evals for a tiny slice of the price. It breaks on hard correctness, so the confidence score is the real trick.
Based on reporting by StackSweep — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Carnegie Mellon researchers tested TypeSafe JEV, a hosted judge that gives only a verdict and label probabilities, and found a narrow but very practical win: on ordinary preference and evidence-grounded factuality tasks, it lands within 3 points of GPT-6 while costing 0.36% as much. The same system falls apart once the job stops looking like a straightforward comparison and starts requiring real correctness checking.
The price gap is brutal. JEV came in at $0.044 per 1,000 judgments, versus $12.18 for GPT-6 Astra, and its median latency was 0.152 seconds rather than 1.885 seconds. Against 16 other judges, including hosted models and local ones, that makes it look less like a fancy chatbot and more like a fast little decision engine built for high-volume triage.
The paper’s results split cleanly by task. On RewardBench, JEV scored 92.2% to GPT-6’s 93.5%. On HaluEval factuality, it was 87.5% versus 86.7%. But on JudgeBench, the gap widened to 78.6% versus 93.1%, and on the style-adversarial RM-Bench hard set it dropped to 74.8% against 94.6%. The authors say the pattern matches the product: it can tell which answer is better supported or more helpful, but it does not re-derive a proof or a calculation to catch a subtle mistake.
That limitation shows up sharply with wrong-but-polished answers. GPT-6 fixed 60 of JEV’s 75 JudgeBench errors, while JEV fixed only 9 of GPT-6’s 24. So the cheap judge is useful, but only if it knows when to step aside.
That’s where the confidence score matters. When JEV reports a probability of 1.0, it is right 99.1% of the time in the pooled sample. With a threshold of 0.9, a cascade can keep JEV’s verdict on 66% of items and send the rest to GPT-6, reaching 91.3% accuracy versus GPT-6’s 91.7% at 47% of the fee. The savings vary by workload: on RewardBench it even beats GPT-6 at 22% of the cost, while on JudgeBench it still needs GPT-6 more often.
The caveats are not subtle. Confidence collapses on style-adversarial pairs, calibration does not transfer cleanly across tasks, and unsupported labels in HaluEval muddy the score. Still, the practical takeaway is plain: for “which answer is better?” or “is this supported?” checks, a decision-only judge with a confidence gate is good enough to save real money. For math, code, and derivations, pay for the stronger judge.
My take — AI-written commentary, not fact-checked reporting
This is the kind of result that should annoy people who treat every eval like a prestige contest. A cheap judge that knows its limits is more useful than a glossy one that blurts out explanations and still misses the point. The real luxury feature here isn’t reasoning text; it’s knowing when to shut up and escalate.
Read more about this at: StackSweep
