Tactile: Giving Computer-Using Agents Hands and Feet

arXiv cs.AI · 6 hours ago

Researchers introduced Tactile, an open-source tool that improves how AI agents interact with desktop applications by converting UI elements into actionable targets with verification capabilities. Performance on macOSWorld tasks improved from 41.1% to 50.0% success rate with Codex, and gains were consistent across multiple agent types. The framework enables agents to use semantic actions, OCR-grounded coordinates, and explicit verification rather than relying solely on screenshot pixel coordinates.