QIMMA قِمّة ⛰: A Quality-First Arabic LLM Leaderboard

Hugging Face Blog · 2 months ago

QIMMA is a new Arabic language model evaluation leaderboard that validates benchmark quality before running model assessments, addressing fragmentation and quality issues across existing Arabic NLP benchmarks. The platform consolidates 52,000 samples from 14 benchmarks across 7 domains and discarded between 0.2% and 12.3% of samples per benchmark after applying automated and human quality review, with ArabicMMLU losing 436 samples at a 3.1% discard rate. Rankings now reflect genuine Arabic language capability, with Qwen3.5-397B achieving 68.06% average score, though smaller specialized Arabic models outperform larger multilingual models on cultural and linguistic tasks.