Bringing the Artificial Analysis LLM Performance Leaderboard to Hugging Face

There's a decent chance you've picked an LLM based on a benchmark score and then been unpleasantly surprised by how slow or expensive it turned out to be in production. Artificial Analysis has been quietly tracking that gap for a while, and now its leaderboard has a new home on Hugging Face, putting quality, price, and speed for more than 100 serverless API endpoints side by side.

The numbers behind it are refreshed constantly: every endpoint gets tested eight times a day, with the leaderboard showing a 14-day median plus percentile spreads for latency and throughput. Quality is a blended score pulled from things like MMLU, MT-Bench, HumanEval, and Chatbot Arena rankings. Pricing is reported as a blended input/output figure assuming prompts run three times longer than responses. Testing spans six workload combinations, mixing short and long prompts with single or parallel queries, which matters because a model that looks snappy on a 100-token prompt can behave very differently at 10,000 tokens under load.

What the May 2024 snapshot makes obvious is just how chaotic pricing has gotten. Claude 3 Opus and Llama 3 8B sit at opposite ends of a 300x price spread — two orders of magnitude apart for tasks that, depending on the use case, might produce comparable results. Speed varies just as wildly, and the market keeps moving faster: when Meta shipped Llama 3, seven different providers had it live within 48 hours, a sign of how aggressively hosting companies are now competing to serve fresh open models.

Artificial Analysis groups the field into three rough tiers. GPT-4 Turbo and Claude 3 Opus sit at the top, expensive and slower but high quality. Llama 3 70B, Mixtral 8x22B, Command R+, Gemini 1.5 Pro and DBRX occupy a middle ground on all three metrics. And then there's the cheap, fast tier — Llama 3 8B, Claude 3 Haiku, Mixtral 8x7B — that looks weak on paper but opens up different system designs.

That last point is the more interesting one buried in the release. Their example: instead of sending GPT-4 Turbo to fetch and read a handful of news articles, you could have Llama 3 8B skim dozens of pages in parallel and hand only the distilled highlights to GPT-4 Turbo for the final summary. Cheaper, and potentially more thorough, since you're covering ten times the source material for less money. It's a reminder that raw model quality is only one input into a system's actual output — architecture and orchestration can matter just as much.