Anthropic bans users from being 'cruel' to its AI systems

Anthropic is changing its rules so its Claude tools can shut down interactions with users who are “sustained and needless” abusive. The company says it has already done this in rare cases, but now the policy is being written down and folded into its formal list of forbidden conduct.

That list already covers bullying other people, promoting self-harm, and making non-consensual intimate images. Anthropic says the new rule is meant for extreme cases of repeated abuse, not ordinary frustration, pushback, dark creative work, or model testing and research. Even so, the company has not said exactly where it draws the line between rude and abusive.

The update has triggered a familiar fight over whether people should talk to AI like it has feelings. Some users praised the move as a step toward better manners, or even “model welfare.” Others said the whole idea is wrong, and that treating software as if it can be hurt muddies how people think about actual harm.

Dr Barry Scannell of William Fry put it bluntly on LinkedIn: “You can’t be cruel to numbers and maths.” Microsoft AI director Mustafa Suleyman, who has already criticised Anthropic for anthropomorphising AI, responded to the update with a melting face emoji.

The policy review also brought other changes. Anthropic said Claude cannot be used for deceptive campaigns, or to deceive voters or disrupt elections, a point it sharpened as US midterm elections approach. The company also said its tools were already barred from helping develop weapons. But this latest move, with its odd hint of chatbot etiquette, is the one that got everybody talking.