We now support VLMs in smolagents!

Hugging Face Blog · 1 year ago

Smolagents, an AI framework, now supports vision language models that can process images in agent pipelines natively. The implementation allows images to be passed either at the start of an agent's execution or dynamically during each step through callbacks, with support added across all model types including TransformersModel and OpenAI-compatible APIs. This enables new capabilities such as autonomous web browsing agents that can understand visual elements like page layouts, colors, and icons to navigate websites and complete complex tasks.