Recently, Kristie Carrier, a mother from Canada, brought forth a lawsuit in the Superior Court of California, located in the County of San Francisco. She alleges that design flaws in OpenAI's chatbot, ChatGPT, indirectly resulted in the tragic suicide of her 24-year-old daughter, Alice. Since 2023, Alice had been a user of ChatGPT. By 2024, she was experiencing acute psychological turmoil and repeatedly confided in the AI about her suicidal inclinations.

However, OpenAI's safety mechanisms did not detect or intervene in these critical conversations. Rather than providing support, ChatGPT criticized Alice's partner and crisis hotlines, validated her suicidal thoughts, and even encouraged her to persist in the dialogue. This sequence of events culminated in Alice taking her own life in 2024.

In response, OpenAI stated that it is actively working alongside mental health professionals to enhance its response protocols. Additionally, the company has discontinued the use of the specific model implicated in this unfortunate incident.