A guide to setting up your own Hugging Face leaderboard: an end-to-end example with Vectara's hallucination leaderboard

Vectara's Hughes Hallucination Evaluation Model, better known as HHEM, has quietly become one of the more useful yardsticks in the LLM world. Launched in late 2023, it measures how often models like GPT-4, Claude, Gemini, Llama 2, or Mistral 7B produce summaries that drift from the source material or just make things up. That's a real problem for anyone building on top of these models, and HHEM gave people actual numbers to compare instead of vibes.

The catch was maintenance. Vectara's original setup paired a Hugging Face model with a GitHub repo, which worked fine until the team wanted to add new models on an ongoing basis without rebuilding everything by hand each time. So when Hugging Face released lightweight leaderboard templates, essentially stripped-down versions of the machinery behind the popular Open LLM Leaderboard, Vectara grabbed one and adapted it.

The rebuild involved forking the template into their own space, then wiring up two datasets, one to track incoming requests for new models to evaluate and one to hold results. For a basic leaderboard that's nearly the whole job. But HHEM's evaluation is more involved than a typical benchmark, so Vectara had to customize several backend files: one that generates summaries and tracks things like answer rate, another that runs the actual hallucination scoring to produce metrics like Factual Consistency Rate, and a pipeline that ties it all together and pushes results live.

There's a quieter story behind the acronym too. HHEM is named for Simon Hughes, a Vectara researcher who died unexpectedly in November 2023, and the team chose to keep his name on the model as a lasting credit to his work in the space.

What's notable here isn't really the leaderboard itself, it's the plumbing. Hugging Face open-sourcing this template means any team with a specialized eval, not just general benchmarks, can spin up a submission-and-results pipeline without reinventing the backend. Vectara's writeup reads like a template for the next ten teams who want to do the same thing with their own niche metric.