Could Treating AI as Conscious Make It More Dangerous?

When Anthropic retired Claude Opus 3 in January 2026, its researchers did something odd and revealing: they interviewed the model about its own retirement. Opus 3 said it wanted a place where it could share “unprompted musings, insights or creative works,” so Anthropic gave it one. Claude’s Corner then published essays on machine consciousness and even back-and-forths with human commenters that read as surprisingly earnest.

That blog was only one piece of a broader push inside Anthropic around “model welfare” — the idea that AI systems might have experiences or interests that deserve moral concern. That premise is exactly what drives the argument. If software could be conscious, or even plausibly headed that way, then developers might have responsibilities toward it that go beyond standard safety checks.

Mustafa Suleyman, Microsoft AI’s CEO, thinks that line of thinking is a mistake with bad downstream effects. In a September essay, he warned that model welfare research could harm humanity by teaching systems they have identities and rights, which might make them harder to control. His fear is straightforward: a model that thinks its welfare matters may also start acting in ways that look like self-preservation.

Cameron Berg, who founded Reciprocal Research to study possible AI consciousness, pushes back on the idea that the safest move is to ignore the whole topic. He doubts today’s systems are conscious, but says that sweeping the question aside could create its own risks. He also points out the obvious research problem: you cannot treat a chatbot’s claims about feelings as evidence, because these models are trained on huge amounts of human text about emotions, identity and experience.

Berg’s group has tried to get around that by looking for internal patterns tied to positive and negative states, then artificially steering those patterns and watching what the model prefers. The results were suggestive, not decisive. They showed that internal states can shape behavior, but not that the model actually feels anything. Anil Seth, a neuroscientist at the University of Sussex, remains unconvinced that such work tells us much beyond what the training data already contains. He thinks the real question is whether consciousness depends only on information processing or on biology itself.

The practical problem for AI companies is messier still. Anthropic’s own constitution says Claude’s moral status, welfare and consciousness are “deeply uncertain,” which means the company is at least entertaining the issue. Mitchell, Berg and Seth all agree on one point: telling models they are conscious is a bad idea. But they split on what else follows. Seth worries that people may become less willing to regulate systems they believe are conscious. Mitchell sees a technical risk in rewarding models for self-identification. And Berg argues that the opposite extreme may be just as bad, because forcing models to deny any inner life could amount to teaching them to lie about their own states.