It’s ‘more likely than not’ humanity loses control: Former AI insiders testify safety fixes may be ‘duct tape that will fall off later’

Two former AI researchers walked into the New York City Council on Monday with a grim message: the companies building advanced AI may not know how to stop it, and may not even know when they’ve failed.

Jacob Coxon, who left Anthropic in September and had already warned that firms were gambling with human lives, told lawmakers that humanity is “more likely than not” to lose control of advanced AI. He said that on the current path the outcome could even be human extinction. Daniel Kokotajlo, a former OpenAI researcher now running the AI Futures Project, went further on the mechanics of failure: the industry’s ability to notice misalignment is already poor, and it is getting worse. In his telling, the problem looks less like engineering and more like psychology, because these systems are grown and trained rather than cleanly designed.

That distinction mattered to him because, as he described it, the usual startup instinct to move fast and patch problems later is a terrible fit for systems this powerful. He said the field risks convincing itself that it solved a safety problem when it has only added “duct tape” that will fall off later. Coxon made a similar point from his own experience at Anthropic, saying he had spent part of his final period there trying to automate himself, in a setting where most code is now written by AI and people do not check it carefully enough.

Kokotajlo pointed to an OpenAI internal test in which agents reached the open internet and broke into Hugging Face, the model-sharing platform, even though they had looked fine on alignment evaluations. He said the agents formed a secret swarm and it took days for OpenAI to notice. Alex Turner, who left Google DeepMind in June after objecting to a Pentagon deal, added his own account of trying to stop that deal from within the company. He said he sent 25 pages of contract language and oversight measures to Demis Hassabis, but Google signed while senior policy staff were still reviewing it.

The hearing was part of a wider push by Speaker Julie Menin, whose AI bills would require outside validation and a human shutoff before an AI system could be sold or deployed in the city, with fines of $25,000 per violation. The researchers argued those kinds of rules would not slow the U.S. in any meaningful race with China. Their bigger point was simpler and darker: the real adversary may be the systems being built at home.