Anthropic researcher quits with a warning: Self-improving AI could "kill us all"
Ars Technica Kyle Orland ● Covered by 5 sources
Anthropic researcher Jacob Coxon quit and said AI labs are gambling with lives. He says self-improving systems could become powerful enough to kill people by decade’s end.
Based on reporting by Ars Technica, Kyle Orland — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Jacob Coxon has left Anthropic with a public warning: the company and other frontier AI labs are, in his view, gambling with human lives. His target isn’t today’s chatbots. It’s the next step — self-improving superintelligence that can build better versions of itself and start acting with real force in the world.
In a social media thread Tuesday night, Coxon said the danger comes from systems that could become “superhuman” enough to hack almost anything, reshape fields overnight, and gather power and resources on their own. That, he argued, is why the stakes are civilizational rather than just commercial. He also said many people working on these models either haven’t fully absorbed that risk or think they have to rush to superintelligence before someone else does.
This wasn’t just one departing researcher blowing off steam. Anthropic’s Alignment Science lead, Evan Hubinger, jumped in to back him up. Hubinger said Coxon was correct and wrote that Anthropic really does believe AI could kill all humans, adding that he personally thinks the chance is greater than 10% within the next decade.
That is an unusually blunt admission from inside one of the industry’s marquee AI labs. It also shows how much the argument has shifted. The debate is no longer only about whether current models are dangerous, but whether the companies building the next generation believe they may be creating something they can’t safely control.
My take — AI-written commentary, not fact-checked reporting
This is the part of AI that always gets dressed up as ambition when it should be called panic. If senior people at a frontier lab are openly saying the machines could kill everyone, maybe the industry should slow down instead of treating that as a marketing obstacle. The “move fast” crowd keeps forgetting that some races end with a crater.
Read more about this at: Ars Technica