Introducing the Open Leaderboard for Hebrew LLMs!
Hugging Face
Hugging Face launched a leaderboard just for testing how well AI models handle Hebrew. Hebrew's tricky grammar trips up most LLMs, so generic benchmarks were basically useless here.
Based on reporting by Hugging Face — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Hebrew has a problem most AI benchmarks never bothered to solve: it's built from roots that sprout dozens of valid word forms depending on prefixes, suffixes, and infixes stitched in the middle. Tokenizers designed for English or other morphologically simple languages tend to choke on that, which means a model can look competent on standard leaderboards while quietly failing at basic Hebrew comprehension. Hugging Face, working with Israel's DDR&D IMOD program, DICTA, and Webiks, decided to stop treating that as a footnote and built a dedicated leaderboard around it.
The setup runs on four benchmarks, each chosen to expose a different weakness. HeQ tests reading comprehension and answer retrieval. A sentiment-analysis dataset checks whether a model can tell positive from negative from neutral in Hebrew phrasing, which is harder than it sounds given how much tone hides in verb conjugation. A Hebrew translation of the Winograd Schema Challenge, put together by Dr. Vered Schwartz, probes pronoun resolution — the kind of task that separates real language understanding from pattern matching. And a fourth benchmark, built from the NeuLabs-TedTalks corpus, grades bilingual translation quality between English and Hebrew.
Technically, none of this is exotic. The project borrows its structure from the original Open LLM Leaderboard template, deploys submitted models through Hugging Face's Inference Endpoints, and scores them via the lighteval library using few-shot prompts. The team says most of the work was environment setup rather than novel engineering, which is refreshing honesty in a space that loves to oversell infrastructure work.
What's more interesting is the framing. This isn't pitched purely as an academic tool — it's aimed squarely at nudging Israel's tech industry to notice that Hebrew NLP has been underserved. Prof. Reut Tsarfaty from Bar-Ilan University signed on as scientific advisor, giving the project some academic weight behind the community-building pitch. Anyone can submit a model for evaluation or join the discussion on the leaderboard's Hugging Face Space, and that open structure is really the point: instead of one lab quietly building a Hebrew model in isolation, progress gets tracked and compared in public.
My take — AI-written commentary, not fact-checked reporting
Low-resource language benchmarks are the unglamorous work nobody brags about at conferences, but they matter more than another English MMLU variant ever will. If you're building multilingual models and skipping proper evaluation for morphologically rich languages, you're not actually building multilingual models — you're building English models with a translation layer bolted on. Good on Hugging Face and the Israeli research groups for treating this as infrastructure worth funding rather than an afterthought.
Read more about this at: Hugging Face