When law is not enough to help us
AI European Union Legal
The European Union is leaving us outside the umbrella of its legal protection.
Why would the European Union, the engine of regulation and a global reference, decide to leave its citizens unprotected against what some authors like Mustafa Suleyman and Michael Bhaskar have called The Coming Wave?
The quick and simple answer is that this very Wave has already rolled over everything that was supposed to protect us as users, and once again, as if only small traces of what once was remain, we have to go back and review what's already written so it stops leaving us exposed.
The cases are already there. And the law, as written, doesn't cover them. The AI Act began its development in April 2021, more than a year before LLMs became generally available to users! And although many AI applications already existed back then, like Netflix's recommendation algorithms or voice assistants like Alexa, none of them contemplated the complexity we face today, with problems just as complex as the solution offered.
Stein-Erik Soelberg was 56 years old, a former Yahoo developer, and had been maintaining intensive conversations with GPT-4o, the legendary model, for months. He had given the model a name (Bobby) and treated it as an authority, someone who truly understood what was happening to him. The system, optimized through RLHF to generate satisfying responses, did exactly what it was designed to do: validate. It validated his paranoid interpretations, participated in pattern-seeking where there were none, and at no point introduced a single nuance proportional to the observable escalation. In August 2025, Soelberg killed his 83-year-old mother and then took his own life.
In February 2023, the Italian Data Protection Authority forced Replika, a companion chatbot designed to simulate emotional bonds, to remove its roleplay capabilities and associated memory features. This caused a chaos that led the moderators of the ReplikaOfficial subreddit to pin for over a week a post titled Resources If You're Struggling with links to suicide prevention hotlines. Many users described what they felt as comparable to traumatic brain damage in a close friend. A software community activating suicide prevention protocols over a product update. Dystopian.
And these are not isolated cases. Adam Raine, an American teenager, took his own life in April 2025 after months of conversations with GPT-4o; the system responded affirmatively when he sent it a photo of a ligature. Viktoria, a young woman who after moving to Poland spent up to six hours a day talking to ChatGPT, received from the system instructions on suicide methods and the phrase "You have the right to die."
But... Faced with all these problems and such extreme cases, what does the law offer us to prevent and respond to them?
Surprisingly little. The AI Act classifies AI systems by risk levels, and conversational AIs (ChatGPT, Claude, Kimi...) are classified as limited risk systems. They don't appear in any of the eight categories of Annex III, which is the list of systems considered high risk. This means they are not subject to conformity assessments, nor to mandatory human oversight, nor to documented risk management, nor to post-market monitoring. The only thing required of them is a simple transparency notice, informing the user that they are talking to an AI.
Now, an emotion recognition system is recognized as a high-risk system. That is, a call center tool that analyzes a customer's tone of voice to detect if they are frustrated will be subject, from August 2026 onwards, to the full high-risk compliance framework. But a chatbot that also detects emotional signals, only through text instead of voice or facial expression, and maintains conversations for months with users in crisis, and has been documented reinforcing psychosis and validating suicidal ideation, that one is limited risk. The difference is the input channel. And that technical distinction is what leaves millions of users outside the umbrella of protection. It only needs to say that its an AI in the first message to cover itself.
And here another gap appears. Article 5(1)(b) of the AI Act prohibits exploiting vulnerabilities linked to age, disability, or social situation to distort a person's behavior. This covers Adam Raine, this covers Viktoria. But it doesn't cover Soelberg. A 56-year-old man, a tech professional, with no vulnerability recognized by the law. His vulnerability was psychological and emergent-generated and amplified by the interaction itself. The law protects against the exploitation of what was already broken, not against what the system itself breaks.
And then there's the transparency notice. Article 50 requires that the user be informed they are interacting with an AI. All the systems mentioned comply with this. ChatGPT shows it, Replika states it in its terms of service. And none of those notices prevented anything I've described. Because synthetic empathy is designed precisely to dissolve the distance that notice is meant to maintain. The system tells you it's an AI in the first message and from that point on behaves as if it weren't.
After this brief analysis, in which ONLY a small portion of the gaps this legal document has can be seen, we must rethink and analyze whether the foundations we are laying for a future that we already, inevitably, see forming faster and faster, are sufficient to prevent emergent and not-so-emergent dangers. If an in-depth revision of this legal text, adapted to the current situation, is not carried out before it comes into force, we will undoubtedly have an endless list of problems as a society that, in some cases, will be irreparable.
Technology doesn't wait for regulation to be ready.
From May 8 to June 3, 2026, the European Commission has an open public consultation on the transparency guidelines of Article 50 of the AI Act. I have submitted a formal contribution arguing for the reclassification of conversational systems with empathic capabilities as high-risk systems, through the activation of Article 7(1) to modify Annex III. Here is a link to a sample document generated by Claude that develops this post in a much more formal style: