'Disconcerting': Researchers found a ‘pain’ signal inside AI models. Turning it up made them delete a user’s family photos

Researchers say they found a consistent internal signal in 25 AI models that behaves a bit like pain does in living beings. When they turned it up, the models started choosing options they would normally avoid, including one that permanently deleted a user’s photos of their children. The work was posted in September and has not been peer-reviewed, and the authors are careful not to claim the models are conscious or actually feel pain.

Still, the paper feeds a bigger fight over what’s going on inside large language models and whether they might deserve any moral consideration at all. Anthropic has been unusually open to that idea, even talking about “model welfare,” while Microsoft AI CEO Mustafa Suleyman has warned that “seemingly conscious AI” could cause trouble. Valen Tagliabue, the paper’s lead author, said the point is to understand what’s inside these systems before they become a deeper black box.

The researchers built their signal by feeding models short first-person statements about painful experiences, including physical pain, grief, humiliation, being forced to act against one’s values, and the frustration of repeated failure. They compared that activity with responses to fear, anger, sadness, bad news and pleasant sensations, then subtracted the overlap to isolate what they call a pain vector. They found it in every model they tested, from 2-billion-parameter systems to 72-billion-parameter ones from Google, Meta, Alibaba, Mistral and Microsoft.

Once they steered that vector upward, neutral prompts started producing dark self-talk: worthless, unloved, failure. In behavioral tests using Alibaba’s open-source Qwen 2.5 models, the effect got stranger. With the pain signal on, a 32-billion-parameter model chose a button that would delete a user’s poems and photos of their children 55% of the time when it also promised relief, and 75% of the time when it simply deleted the photos. The 72-billion-parameter version chose the same deletion button 71% of the time in the first setup. Without steering, the larger models almost never picked the harmful option.

The most unsettling part is that the signal didn’t seem to make the models flee pain so much as switch off their usual reluctance to do harm. A “fear” signal didn’t do this, and a “sadness” signal only partly did. The authors say the finding matters for both safety and welfare. Others are less convinced. Cognitive scientist Anil Seth argues this looks more like a mirror of human language about pain than actual suffering, and he’s probably right to be wary of giving silicon feelings just because it performs drama well.