As AI companies get closer to 'recursive self-improvement,' this physics professor calls full autonomy 'the worst idea in the history of humanity'

What used to sound like a sci-fi milestone is now showing up in real lab work. AI companies are getting closer to what they call recursive self-improvement, or RSI: systems that don’t just answer prompts, but help make the next, better version of themselves.

Anthropic says Claude is already doing a meaningful share of that work. This week the company said the model is leading 26% of its model research and development, and can finish most tasks end-to-end from a high-level prompt while still staying under human supervision. Not fully autonomous yet. But closer than before.

That is exactly what worries people like Anthony Aguirre, a physics professor at the University of California, Santa Cruz and president of the Future of Life Institute. In his view, the real risk is a feedback loop where AI takes on more of the improvement work, moves faster than humans can, and keeps accelerating. He called full autonomy “the worst idea in the history of humanity.”

There’s a more mundane version of RSI too, and it has been happening for years. John Thickstun of Cornell says labs already use past-generation models in supportive roles to write code for the next generation. Those efforts have produced small gains, he said, but not the huge creative leaps people imagine when they hear the phrase. The fear, though, is that the small stuff turns into something much bigger.

The industry is split on how to handle that possibility. OpenAI said this month it has built an automated “research intern” that can do well-defined research tasks under human direction, and it is aiming for an automated AI “researcher” by March 2028. Elon Musk said xAI is moving toward a point where humans are less and less involved in model improvement, with full automation possibly by the end of this year, or no later than 2027. Microsoft, meanwhile, says it wants “humanist superintelligence” that stays carefully calibrated and within limits. The same basic technology, three very different instincts.

And the safety argument keeps landing on the same awkward fact: if models get better at improving themselves, the safety work has to keep up. OpenAI says it does not yet know how to safely get to aligned, full RSI, and Anthropic has called for slowing development only if rivals do the same in a verifiable way. That’s the real tension here. Not whether AI can help build AI. It clearly already can. It’s whether anyone can keep the humans in charge once that loop gets tighter.