PaperBench: Evaluating AI’s Ability to Replicate AI Research

OpenAI Blog · 1 year ago

Researchers created PaperBench, a benchmark for testing whether AI agents can reproduce published AI research papers. The benchmark measures performance across tasks like implementing algorithms, running experiments, and validating results from academic publications. This allows evaluation of AI systems' capability to autonomously conduct scientific work rather than just answer questions about it.