MAG: A Web-Agent Benchmark and Harness for Multimodal Action and Guide Generation

arXiv cs.CL · 18 hours ago

MAG is a benchmark that combines web task execution with automated guide generation, grounding agent actions in screenshots rather than DOM trees. The benchmark includes a training harness with LLM-assisted annotation and a GRPO training method that increased task success from 6.9% to 13.2% on a 9B parameter model. Despite these improvements, the strongest evaluated model completes fewer than 40% of tasks, indicating significant remaining challenges for multimodal web agents.