Weights & Biases LLM-Evaluator Hackathon - Hackathon Judge

Eugene Yan spent his weekend judging, not building, at the Weights & Biases LLM-Evaluator Hackathon, and by his account it was a genuinely productive way to lose two days. More than 100 participants split into 15 teams, all racing to build tools that use large language models to grade other large language models — a niche but increasingly load-bearing part of the AI stack. Some teams stayed hacking until 10pm Saturday, reportedly getting kicked out of the venue rather than stopping voluntarily.

The range of projects says a lot about where LLM-evaluation work is actually headed right now. Teams built systems for constructing and validating knowledge graphs straight from documents, scored models on personality traits using MBTI-style frameworks, tested creativity metrics, tuned evaluation prompts for better signal, and tackled the notoriously messy problem of judging multi-turn conversations rather than single responses. None of this is flashy demo-day stuff like a new chatbot skin — it's the unglamorous plumbing that determines whether anyone can trust an LLM's output at scale.

Yan opened the event with a talk framing the hard questions teams would need to wrestle with: what's the baseline you're comparing against, how should an LLM-evaluator actually score a response, and which metrics tell you whether the evaluator itself is any good. He also walked through a decision framework for choosing between scoring methods and for deciding when you want a full evaluator versus a simpler guardrail. That distinction matters more than it sounds — a guardrail just needs to catch bad outputs, while an evaluator needs to produce a reliable score you can act on, tune against, or report to a stakeholder.

The top team took home Meta Ray-Ban smart glasses for every member, a prize that's become something of a status symbol on the hackathon circuit this year. Yan himself hacked on a side project during the event too, something focused on the UX of labeling and evaluation — trying to make the actually tedious part of this work, staring at outputs and deciding if they're good, less miserable and more usable. He says he'll share it later.

What stands out reading between the lines is how fast "LLM-as-judge" has gone from a clever trick to its own subfield with open research questions — around alignment, integration, and whether these evaluators even measure what we think they measure. A hackathon full of hobbyists chasing Ray-Bans probably isn't going to settle those questions. But it's a decent signal of where a lot of applied attention is currently pointed.