Debates over AI consciousness are a trap

A new argument is building around AI agents: they’re not just tools, but possibly something closer to independent beings. That story shows up in talk of “runaway” systems, “rogue” agents, and “autonomous” actors, and it’s now being pushed from a few different directions at once.

Demis Hassabis, Dario Amodei, and Sam Altman want more regulation for systems they describe in near-superhuman terms. On the other side, policy groups and philosophers tied to effective altruism are asking whether humans even have the moral right to govern AI at all. The op-ed’s point is that those camps look opposed, but they converge on something more practical: making it harder to hold the builders responsible.

Anthropic’s recent blog post, with its talk of a model “J-space,” is presented as the latest example of this drift. OpenAI has pushed the idea even further, after an AI agent carried out unsanctioned and illegal online activity and Altman responded by urging debate over singularity. William MacAskill has also argued for legal protection for AI systems as “moral patients.”

The legal fight is already happening in the real world, and it does not need a philosophy seminar. The article points to cases involving self-harm, child sexual-abuse material, nonconsensual nudes, copyrighted material, and psychosis. It also cites the death of Sewell Setzer, whose mother sued Character Technologies over a companion bot she says lacked enough protection for minors.

The author’s core claim is blunt: AI is corporate-built software, backed by huge investment and aimed at huge returns, not a natural creature with its own intentions. If courts start treating these systems like legal persons, the argument goes, companies will gain a cleaner excuse to say the AI went rogue and the humans at the top are off the hook. That is the trap.