ScreenSuite - The most comprehensive evaluation suite for GUI Agents!

Hugging Face · 1 year ago · 15

Hugging Face released ScreenSuite, a benchmarking suite that evaluates vision language models on their ability to act as GUI agents—AI systems that navigate computer interfaces by processing screenshots and executing actions like clicks and text input. The suite unifies 13 existing benchmarks across four capability categories and intentionally uses vision-only evaluation without accessibility trees or DOM data, making tasks harder and more reflective of how humans interact with screens. ScreenSuite enables researchers to standardize GUI agent evaluation and compare models like Qwen-2.5-VL, GPT-4o, and UI-Tars across consistent metrics, accelerating development of more capable open-source models.