Introducing AIMIP: The AI weather and climate model intercomparison project

Weather forecasting got its AI moment years ago, when neural nets trained on the ERA5 reanalysis dataset started beating traditional physics-based models on short-range predictions while using a tiny fraction of the compute. Climate modeling, the decades-and-centuries version of the same problem, has lagged behind. Partly that's because simulating climate is a much harder computational lift, and partly it's because nobody had agreed on how to grade these new AI systems in the first place.

Ai2 is trying to fix the second problem with AIMIP, a model intercomparison project built specifically for AI-driven climate forecasts. The idea borrows from CMIP, the long-running framework that's shaped how physically-based climate models get evaluated for things like greenhouse gas projections. AIMIP's first phase, released May 13, pulled in eight model runs from six groups — Ai2 itself, NVIDIA, Google Research, the University of Washington, the University of Maryland, and the ArchesWeather team. Each model had to forecast the global atmosphere from 1979 to 2024, trained only on data through 2014, with the final decade held out as a genuine test.

The results are a mixed bag, which is honestly more useful than a clean win. On average historical climate patterns, AI models mostly outperformed a conventional physics-based benchmark, with the best ones cutting near-surface temperature error roughly in half. That's a real achievement. But when Ai2's team pushed the models to extrapolate the long-term warming trend into that held-out decade, performance splintered — some models tracked it well, others badly undershot it. Things got worse in a stress test where researchers instantaneously warmed the global ocean surface by 2 or 4 degrees Celsius, a scenario that will never happen in reality but is a useful way to probe how models behave outside their training distribution. Several produced outputs that looked physically implausible.

That gap matters because the whole appeal of AI climate models is speed — up to a thousand times less compute than traditional simulations — which could open climate science to researchers and institutions that never had supercomputer access. But speed is worthless if the model quietly falls apart the moment conditions drift from what it saw in training, which is exactly the situation climate scientists care about most: predicting a future that, by definition, doesn't look like the past.

Ai2 is hosting the Phase 1 dataset through Germany's DKRZ, with a wider release planned via the Earth System Grid Federation, and says future phases could add ocean and sea-ice coupling plus a broader range of emissions scenarios. For now, the project's real contribution isn't a leaderboard winner — it's a shared, public yardstick that didn't exist a year ago, in a field where compute scarcity and closed development had made robust comparison nearly impossible.