Introducing HELMET: Holistically Evaluating Long-context Language Models

Hugging Face · 16 April 2025, 00:00

Researchers introduced HELMET, a benchmark for evaluating long-context language models across diverse real-world tasks like summarization, retrieval-augmented generation, and citation generation. The benchmark tested 59 recent long-context models across input lengths from 8K to 128K tokens, using model-based evaluation metrics instead of traditional n-gram methods. The results show that frontier models degrade significantly on complex tasks with longer inputs, and no single model excels across all task categories, indicating the need for multi-faceted evaluation approaches in developing long-context systems.