We now support VLMs in smolagents!

Hugging Face just gave its smolagents framework something it's been missing since launch: eyes. The library, which builds AI agents around a code-writing ReAct loop, now natively supports vision language models, letting agents look at images and screenshots rather than relying purely on text descriptions of the world.

The motivation is pretty concrete. Web browsing agents, the team argues, have been operating half-blind. A page's layout, its icons, the color of a button telling you something is disabled, none of that survives when you strip a webpage down to raw text. So Hugging Face added two ways to feed images into an agent: a batch method where you hand over a list of images when you call agent.run(), useful for things like parsing long PDFs with charts and diagrams, and a dynamic method using callbacks that fire at the end of every step, which is what makes a live web-browsing agent possible.

Under the hood, every smolagents agent is built on a MultiStepAgent class that cycles through logging, prompting, parsing, and executing actions. The vision update slots a screenshot-capturing callback into that loop's step_callbacks hook, storing images in an ActionStep's observations_images attribute after each move the agent makes. To prove it works, the team built a browser agent using Helium (a Selenium wrapper) paired with Qwen2VL-72B via the Fireworks API, plus a couple of custom tools for dismissing pop-ups and searching page text with Ctrl+F, since those interactions are notoriously fiddly for automation.

The demo task was genuinely tricky: navigate to GitHub's trending repos, find the top author of the top repo, and dig up their commit count over the past year. Hugging Face is upfront that success depends heavily on which VLM you plug in, with stronger models like Qwen2VL-72B or GPT-4o completing it more reliably than smaller ones. TransformersModel users need to remember one setup quirk too, setting flatten_messages_as_text to False, or the vision inputs won't process correctly.

This isn't a finished product so much as a capability unlock. Hugging Face is framing it as a starting point, pointing developers toward the documentation and inviting them to build their own vision-equipped agents, which suggests the more interesting use cases are still ahead rather than already solved.