Procgen and MineRL Competitions

OpenAI just put its name on two new competitions for NeurIPS 2020, and the guest list alone is worth noting. AIcrowd is running the logistics, Carnegie Mellon is bringing the MineRL expertise, and DeepMind — usually a rival in the leaderboard wars — is co-organizing too.

The two testbeds are Procgen Benchmark and MineRL, and they're picked for a reason. Procgen throws agents into procedurally generated game levels, so an agent can't just memorize a fixed map; it has to generalize to layouts it's never seen. MineRL flips the challenge sideways: it's built on Minecraft and leans on human demonstration data, pushing teams to get agents learning useful behavior without needing millions of trial-and-error episodes.

Both benchmarks target the same soft spot in reinforcement learning — sample efficiency and generalization, the two things that make RL painful to deploy outside a game engine. An agent that needs ten million tries to learn a task, or one that falls apart the moment the level looks slightly different, isn't much use in the real world. These competitions are effectively asking research teams to prove they've made progress on that gap, in public, against a shared scoreboard.

What's notable here isn't the tech itself so much as the guest list. OpenAI and DeepMind don't co-organize much of anything, given how often their papers and product announcements read like competing press releases. Pairing them up with an academic anchor like CMU and a competition platform like AIcrowd suggests this is meant to be a genuinely open, comparable benchmark effort rather than a lab flexing its own results.