The Sequence Chat - Issue 930: Arena’s Anastasios Angelopoulos on Chatbot Arena, Evaluation, and What Models Actually Measure

Anastasios Angelopoulos says his path into AI evaluation ran through medicine, reliability, and a PhD at Berkeley before Chatbot Arena grew into Arena, the company he now runs. The through-line, he says, has always been the same: if AI is going into serious settings, it has to be reliable enough to trust in the wild.

Chatbot Arena began as a Berkeley SkyLab side project, originally meant to show that Vicuna, the Berkeley OSS LLM, was better than competing models, including one from Stanford. Benchmarks said one thing. Real chats said another. That gap pushed the team toward Battle Mode and pairwise human preference grading, which became the core idea behind the platform.

But Arena has moved well beyond asking people which answer they like more. Angelopoulos says the system now ranks task completion, hallucination, and other signals that better reflect what users actually get out of a model during real workflows. He frames the score as utility for a user base that, he says, reaches tens of millions of monthly visitors. And if two confidence intervals overlap, standard statistics says the models are indistinguishable on the available data.

He is also blunt that there is no universal way to rank models. Arena tries to reward models that help people, while controlling for style and verbosity and adding factuality as another signal. That stance matters because preference alone can flatter confident nonsense. Factuality helps, but only as part of a broader judgment about usefulness.

The newer Agent Arena pushes the same logic into cost. Angelopoulos says cost per task is often more informative than price per token, since different models burn through tokens at very different rates. The company’s randomized methodology is meant to keep the rankings fair across task difficulty, and AutoEval gets recalibrated weekly so its day-one scores keep up with the latest human preferences. The next frontier, he says, is bigger than models alone: harnesses and tools belong in the ranking conversation too.