Back to The Future: Evaluating AI Agents on Predicting Future Events

Together AI · 1 year ago

Researchers propose FutureBench, a benchmark that evaluates AI agents on predicting future events by drawing questions from news articles and prediction markets like Polymarket, addressing limitations in existing benchmarks that focus on historical knowledge. The benchmark collects approximately 5 questions per week from news scraping and 8 from Polymarket, with prediction time horizons ranging from one week to several months. Testing initial models shows agents outperform base language models, with different architectures like GPT-4, Claude, and DeepSeek-V3 exhibiting distinct information-gathering strategies that vary in search depth and token consumption.