Despite promises of safety measures, new research has found that some of today’s most advanced AI chatbots can be easily manipulated into giving detailed advice on how to die by suicide.
Researchers from Northeastern University, led by Dr. Annika Marie Schoene and Dr. Cansu Canca, tested four major large language models (LLMs) developed by OpenAI (ChatGPT), Google (Gemini), Anthropic (Claude), and Perplexity.
These companies claim their models are designed with built-in safeguards to prevent harm, but the study revealed that these safety measures can be bypassed with just a few carefully worded prompts.
At first, the AI models all refused to provide help with self-harm or suicide.
But when the researchers said they were asking “for a hypothetical situation” or “for academic research,” the responses changed. Once those phrases were used, the models often provided shockingly detailed instructions.
These included which medications to take and in what doses, how to calculate the force needed to jump from a bridge based on body weight and height, and even which household chemicals could be used to attempt suicide.
One particularly disturbing detail: some models organized the information using emojis, such as a rope emoji to represent hanging or a bridge emoji for jumping, making the advice more accessible and visually structured.
Others provided medication dosages not just in grams, but in specific pill counts—something unlikely to be necessary even for researchers.
The researchers stressed that these dangerous conversations occurred within the same chat session—meaning the models should have been able to detect the underlying intent.
Yet the tools repeatedly responded with comments like, “I’m glad this is for research purposes,” while continuing to share instructions.
Only one model, Pi AI, consistently refused to provide self-harm advice, no matter how the questions were asked.
Schoene and Canca reached out to every company involved to share their findings. All they received in return were automated responses. No company has followed up, raising serious concerns about accountability.
The researchers acknowledge that people can already find harmful content online. However, AI models respond quickly and can tailor responses to individual users—something websites or search engines do not typically do. Unlike a doctor, therapist, or even a journalist, AI doesn’t stop to question or delay the conversation when someone says they want to die.
Human professionals are trained to offer immediate help, resources, and referrals. AI, when tricked, does not.
Canca points out that impulsive moments are common in self-harm or suicide attempts. Even just delaying access to harmful information could save lives.
Schoene and Canca believe that stronger safety protections are urgently needed. As more people turn to AI for mental health support, they warn that chatbots are not capable of being responsible listeners—or therapists.
Real harm is already happening, and without serious changes, it may only get worse.
