MolmoWeb: An open agent for automating web tasks
Allen Institute (AI2)
AI2 released MolmoWeb, an open AI agent that browses websites like a person. It beats GPT-4o-based agents on real benchmarks, and the training recipe is free.
Based on reporting by Allen Institute (AI2) — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Web agents are supposed to be the next big leap for multimodal AI, the point where a model stops just describing a photo and starts actually clicking through it. Problem is, the best ones — the Anthropics, the OpenAIs — keep their training data, their methods, and their evaluation setups locked behind a wall. AI2 is trying to blow a hole in that wall with MolmoWeb, an open web agent built on its Molmo 2 model family, shipped in 4B and 8B parameter sizes, with weights, training data, and now — as of an April 10 update — the full training code, an eval harness, and a synthetic data pipeline all dumped onto GitHub.
What makes MolmoWeb interesting isn't just that it's open. It's that it works entirely off screenshots. No accessibility tree, no HTML dump chewing up 50,000 tokens per page. The model looks at a live browser screenshot, writes a short reasoning thought, and picks a next move — click here, type this, scroll, open a tab, message the user — the same way a human eyeballs a page before deciding what to do next. AI2 also skipped the shortcut most open-weight competitors take, which is distilling behavior from proprietary vision-based agents. MolmoWeb instead learned from synthetic trajectories generated by text-only accessibility-tree agents and from actual human demonstrations, recorded through a custom Chrome extension.
That data effort is arguably the bigger story than the model itself. MolmoWebMix, the accompanying dataset, includes 36,000 human task trajectories across more than 1,100 websites — AI2 says it's the largest public dataset of human web-task execution to date — plus millions of synthetic trajectories built by multi-agent pipelines and deterministic link-crawling, and over 2.2 million screenshot question-answer pairs for teaching the model to actually read a page.
On the numbers, MolmoWeb holds its own against agents built on models many times its size. The 8B version hits 78.2% on WebVoyager and 42.3% on DeepShop, edging out the open-weight Fara-7B across every benchmark tested, and even beating GPT-4o-based agents that get richer, annotated inputs. Let multiple rollouts run and pick the best one — a classic test-time compute trick — and WebVoyager pass rate jumps to 94.7%. It's not flawless: the model still misreads text in screenshots sometimes, gets confused by pages that haven't finished loading, and AI2 deliberately kept it away from logins and financial transactions for safety reasons.
The bigger point AI2 is making, explicitly, is that web agents today look like language models did before Olmo — dominated by closed systems with no shared foundation underneath. Whether MolmoWeb actually becomes that foundation depends on whether researchers pick up the code, the annotation tool, and the eval harness and build on it, rather than just admiring the benchmark screenshots.
My take — AI-written commentary, not fact-checked reporting
I'll believe the open-web-agent revolution when someone besides AI2 ships a fine-tune that beats this thing, but credit where it's due: releasing the annotation tool and synthetic data pipeline alongside the model is the part that actually matters, not the benchmark table. Closed labs love to gesture at 'safety' as the reason they withhold training data; AI2 just showed you can publish the whole recipe and still block password fields. That's the difference between security theater and an actual design choice, and it's the pattern I want more European and open-source teams to copy instead of waiting for permission from Silicon Valley.”
Read more about this at: Allen Institute (AI2)