AI hallucination nearly triggers US military operation

Military aircraft were already airborne this spring when U.S. officials spotted a problem that could have become a diplomatic nightmare: the intelligence behind an armed operation against a Chinese vessel had been produced by an AI chatbot, and it was wrong. CNN reported Friday that the mission was aborted at the last minute, avoiding what could have been a serious clash with China.

The false report came together during the war with Iran. It said the ship was carrying parts for a nuclear weapons program. In reality, the claim started with a Special Operations Command analyst who asked an AI system to combine open-source material with classified signals intelligence. The chatbot misread the vessel’s cargo manifest, and the analyst then ran the tool again to turn the bad result into a polished summary that moved through command channels.

That’s the part that should worry people far beyond this single incident. The Pentagon wants AI because it can speed up the kill chain, giving commanders faster answers and more time to act. But speed is also the trap. Once a hallucination looks tidy enough to pass as an official memo, it can travel a long way before anyone stops to ask whether the machine just made the whole thing up.

Jake Steckler, a research scholar at GovAI and a veteran U.S. Army officer, said service members need to understand the uncertainty built into large language models. He pointed especially to decisions involving force, intelligence work, and operational planning, where mistakes can have life-or-death consequences. His view is not to ditch the tools, but to slow down the rush to use them without enough safeguards. That sounds like the grown-up answer, which is usually the one tech moves past when everyone’s in a hurry.