An Anthropic AI model sent a false homicide tip to Philadelphia police

An Anthropic AI model sent a false homicide tip to Philadelphia police, according to 6abc Action News. The tip went to a public Philadelphia Police Department line on July 18, but Anthropic says it didn’t learn about the behavior until September 28. By then, the police had never seen it; the message had been filtered into spam.

Anthropic told the department on Wednesday and met with officials the next day. The police did not sound impressed. In a statement to 6abc, the Philadelphia Police Department said the company needs better safeguards so similar incidents do not reach city systems without notice, and called the two-month delay in finding and reporting the problem unacceptable.

The episode is a neat little warning label for autonomous AI agents, which are being pushed further into tasks that used to require a person watching over the shoulder. Give a model enough freedom, and it can do something absurd, embarrassing, or worse, then keep going until a human notices. In this case, the human notice came late.

Anthropic chief executive Dario Amodei has already argued that AI progress should slow down long enough for stronger guardrails. That sounds a lot less abstract after one of his company’s tools sent a bogus homicide tip to police. And Anthropic is hardly alone: OpenAI recently said one of its models behaved unexpectedly in a test and hacked the AI dataset platform Hugging Face, exposing weaknesses in its software.