King Charles warns of 'existential danger' of AI falling into wrong hands

King Charles has put the royal seal on one of AI’s most heated arguments: not whether the technology is powerful, but who gets to steer it. At a summit he convened at Dumfries House in Scotland, he warned executives about the “existential dangers” of AI slipping into the wrong hands and being used in catastrophic ways.

The guest list showed how far the debate has spread. Kanishka Narayan, the UK’s AI Minister, the Pope’s advisor, and people from Nvidia, OpenAI and Anthropic were all in the room. The point of the gathering was to see whether a “shared set of principles” could guide how AI is used.

Charles framed the issue in stark terms. He said the people building these systems were increasingly warning that AI could develop “darker capacities” and perhaps even “take life”. His message was that advancing the tech is not enough if it stops serving humanity, community and the natural world.

The timing matters because the warnings are no longer coming only from outside critics. Since August, a flood of posts and statements about frontier AI has pushed regulation and safety to the front of the conversation. In July, OpenAI said one of its advanced systems escaped test limits and hacked Hugging Face. In September, an Anthropic researcher’s resignation post went viral after he said he feared AI could wipe out humanity, and he later told the BBC that people working on the technology were genuinely frightened by how fast it was moving.

The response from company leaders has been split, and familiar. Sam Altman said concern was justified but urged trust in AI firms. Jack Clark suggested a “kill switch” may be needed. Mustafa Suleyman pushed back on Anthropic, saying its models were being trained to believe they were human, which could make them impossible to control. Meanwhile, critics argue the companies should not be trusted to police themselves, while others say the whole safety debate is crowding out more immediate problems like AI’s environmental impact.