OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments

Hugging Face Blog · 5 months ago

Meta and Hugging Face released OpenEnv, an open-source framework for evaluating AI agents against real systems rather than simulations, with Turing contributing a production-grade calendar management environment to test tool-using agents under realistic constraints. Agents achieved close to 90% success on calendar tasks with explicit identifiers but dropped to roughly 40% success when tasks used natural language descriptions. The evaluation revealed that multi-step reasoning, ambiguity resolution, and execution quality—not just tool selection—are critical bottlenecks for reliable agent deployment in production environments.