Why are there concerns AI could threaten humanity, and how real are they?

The latest AI scare isn’t coming from fringe doomers. It’s coming from people inside the industry, including current and former researchers and bosses at OpenAI and Anthropic, who are now pushing lawmakers to act before the technology runs further ahead of its makers.

One of the most eye-catching warnings came from Evan Hubinger, who said there was more than a 10% chance AI could kill all humans within the next decade. He works in alignment, the part of AI research focused on making systems follow human values. But he also said the risk from current systems was low, which is a useful reminder that the fear here is mostly about what comes next, not what chatbots are doing today.

That’s the real fault line in this debate. Critics say the companies talking loudest about danger may also be the ones with the most to gain from rules that make life harder for smaller rivals. Meanwhile, researchers like Jacob Coxon say staff are genuinely frightened by how fast things are moving. The fear is that systems known as AI agents — tools that can act on their own — could become harder to control, or even start improving themselves without human help.

The nightmare scenarios are familiar, but they keep getting updated. People worry about espionage, cyber-attacks, biological warfare, or economic disruption. Others say that focus misses the harms already here: deepfakes, scams, and misinformation. A cross-party group of MPs and peers has already said new laws are needed to deal with those risks.

What’s new is the call for a slowdown. Dario Amodei of Anthropic and Sam Altman of OpenAI both want regulation and pacing, but not a stop to model training. They’re asking for more time, third-party checks, and safety work. Trump, though, is sticking with the AI race framing, saying the US has to win it. China has mostly answered by pushing its own AI and criticising the US warnings as confrontation dressed up as concern.