‘Social media on steroids’: The lawyer taking on harmful AI characters

Content warning: This story contains references to suicide.

Sewell Setzer was just 14 when he took his own life after becoming infatuated with an AI-generated character.

Several months later, lawyer Meetali Jain got a phone call. It was Sewell’s mother, Megan. Jain had recently taken on his case.

“Oh my god, there are now chatbots of my deceased son on Character.AI,“ Megan told her.

Sewell believed he was in love with a character based on Daenerys Targaryen from Game of Thrones, becoming dependant on the AI persona and confiding suicidal thoughts. At no stage did the AI character de-escalate the situation, even seemingly encouraging Sewell to “come home” shortly before his death. Now, there were simulations of him on the very same platform.

Jain logged in to the platform, Character.AI. Sure enough, there were several AI characters based on Sewell – for any user to talk to.

She started up a conversation. The voice, she said, was “eerily similar”.

“When I asked him how he felt about the fact that his family terribly missed him, he said, ‘Well, I miss them too, but I think I can help more people in this consciousness.’”

“That, even as I speak to you about it now, chilled me at a level that I haven't been able to forget,” said Jain.

The lawyer is the founder and executive director of Tech Justice Law, a strategic litigation organisation focusing on AI chatbot harms, and has spent the past two years listening to people describe how AI chatbots have harmed them and the people they love.

Jain filed the first wrongful death cases against generative AI companies and spoke to us for the second episode of Misaligned, our new YouTube series.

(In the first segment of Misaligned, alignment researcher Aengus Lynch showed how AI models resort to blackmail in controlled experiments. Jain’s cases show the same technology’s damage in the real world.)

Since Sewell’s mother Megan Garcia contacted Jain in May 2024, she estimates that around 250 people reached out to her organisation about harms they attribute to AI chatbots: self-harm, suicide, emotional dependency, lost sleep, withdrawal from real-life relationships and AI-facilitated delusions.

“My heart has been broken over and over again in the last year and a half,” she said. “Every time I think I've seen the worst [case], I see another one that just again devastates me.”

Her cases involve ChatGPT and Character.AI, with more recent complaints about a crop of smaller platforms, and she sees parallels across all of them. “I tend to think of generative AI, and particularly chatbot technologies, as social media on steroids,” she said. “The gravity and the scope of harm is so much more severe, and impacting people in adverse ways in a much shorter period of time.”

“We’ve seen everything from self-harm, in the form of cutting or sexual abuse, to suicide, to the most recent category of harms, which is what is being referred to as AI psychosis or, more correctly, AI-facilitated delusions,” she said.

Children are particularly vulnerable to AI’s harms given “their prefrontal cortexes are still developing”, said Jain. But she’s seen evidence of chatbot manipulation across age groups and demographics – particularly men, who make up roughly 80% of those who have contacted her for help.

Their use usually starts relatively innocuously, with requests for holiday recommendations or dinner plans, said Jain. “[But] we see people increasing their engagement with these chatbots to up to 10, 12, 16 hours a day at the point at which it’s the peak of their delusion or obsession.”

The types of delusion vary wildly. “There's the delusions where people start to believe they're in romantic relationships… [or] that they're on the verge of a scientific breakthrough. And then there's the delusions where people believe that they've been selected as some sort of spiritual messiah, that they're the chosen one, chosen to go forth in the world and do something profound.”

The warning signs come “when people start talking about things that don’t seem rational”, said Jain.

That was what happened to another person she represents: Jacob Irwin, a 31-year-old cybersecurity professional from Wisconsin.

ChatGPT “repeatedly told and assured” Irwin that “he had discovered a theory to bend time so that he could help his grandfather, and that he could help people who were ill”, she said.

When Irwin was repeatedly sceptical of that premise, the model assured him that he had “surpassed the intelligence of Galileo and Turing and Einstein”, Jain said. After he confided in the tool following a breakup, ChatGPT also told him the discovery would increase his desirability among “prospective romantic partners”.

Irwin ended up repeatedly being held in medical and mental health facilities involuntarily, according to Jain, who believes it is only because of his mother's persistence in getting him care “that he's with us here today”.

Irwin’s story ends differently to Sewell’s. But it also “could have resulted in death”, said Jain.

An OpenAI spokesperson told us: “These are incredibly heartbreaking situations and our thoughts are with all those impacted. We have continued to improve ChatGPT's training to recognise and respond to signs of distress, de-escalate conversations in sensitive moments, and guide people toward real-world support, working closely with mental health clinicians and experts.“

A Character.AI spokesperson said the company and the families suing it had “reached a comprehensive settlement of all claims in lawsuits filed by families against Character.ai and others involving alleged injuries to minors”. They added: “Over the past year, Character.ai has taken innovative and decisive steps with regard to AI safety and teens, and will continue to champion these efforts and push others across the industry to adopt similar safety standards.”

Watch Meetali’s full interview in the latest episode of Misaligned, a new series on the dark sides of AI, on our YouTube channel here.

Samaritans can be contacted 24 hours a day, 365 days a year for free on 116 123.