Community Evals: Because we're done trusting black-box leaderboards over the community

Hugging Face · 8 months ago · 21

Hugging Face launched a decentralized evaluation system where benchmark datasets can host leaderboards and any community member can submit model evaluation results via pull requests. The initial rollout includes four benchmarks—MMLU-Pro, GPQA, and HLE among them—with results stored as YAML files in model repositories and aggregated automatically across the Hub. This creates a transparent record of evaluation sources and methodology, allowing the community to track and build upon scores rather than relying on conflicting reports from papers and closed platforms.