AI Therapy under the EU AI Act

Millions of people now vent to chatbots instead of therapists, and regulators in Brussels have finally caught up to that fact. A new breakdown of the EU AI Act lays out exactly how general-purpose AI systems—think ChatGPT, Gemini, Claude—get treated when someone uses them for mental health support instead of, say, drafting an email. The short version: it depends entirely on what the system does and who it's talking to, and the rules bite differently at the system level versus the underlying model level.

At the harshest end, Article 5(1)(b) flatly bans AI systems that exploit vulnerabilities tied to age, disability, or difficult circumstances in ways that cause real harm. The Commission's own guidelines give an unsettling example: a mental-health chatbot that nudges someone with cognitive impairments into buying overpriced medical products. Crucially, intent doesn't matter here. If a system's effect is to manipulate a vulnerable teenager or an isolated elderly user into harmful behavior, it can be banned even if the company never meant for that to happen.

Short of an outright ban, a chatbot might still land in high-risk territory if it counts as a medical device under the EU's Medical Device Regulation—say, something explicitly designed to monitor or manage depression rather than just support general wellbeing. That triggers a heavier compliance load: risk management systems, technical documentation, human oversight, accuracy and cybersecurity standards, the works. Even systems that dodge that classification still owe users basic transparency—they have to make clear they're an AI, especially for vulnerable users, and a one-time disclaimer at the start of a chat may not cut it if someone's getting emotionally attached over repeated sessions.

Then there's the model layer, which is arguably the more consequential piece for something like OpenAI or Anthropic. Providers of GPAI models with systemic risk have to actively identify and mitigate risks to public mental health, and they're required to report serious incidents where their model contributed to serious mental or physical harm. That obligation already kicked in back on August 2, 2025. Combine that with enforcement starting August 2026, and the pressure on model builders to think about psychological harm isn't hypothetical anymore—it's a compliance deadline with teeth, even if nobody quite knows yet how strictly it'll be enforced.