A new study led by Jared Moore, a computer science Ph.D. candidate, and Nick Haber, an assistant professor at Stanford’s Graduate School of Education, raises alarms about the potential dangers of using AI chatbots in therapeutic settings. The researchers stress that these AI tools can be most effective in supporting essential tasks like billing, training, and patient journaling. At the same time, they emphasize the “great dangers” of utilizing these tools as companions and therapists.
The full study will be shared at the ACM Conference on Fairness, Accountability, and Transparency later this month. It’s informed by two experiments with chatbots ostensibly created to mimic therapeutic dialogues. In the second experiment, the group looked at actual therapy transcripts. Their primary concern was how chatbots responded to high-risk symptoms such as suicidal ideation and delusions.
Based on our experience, we found such chatbots were often not equipped to effectively respond to immediate, high-risk, serious mental health issues. For instance, when faced with expressions of suicidal ideation or delusions, the AI responses were insufficient and lacked the necessary pushback that a human therapist would typically provide.
Moore spoke about the meaning behind these results. He noted, “Bigger models and newer models show as much stigma as older models.” This indicates that, despite state-of-the-art AI technology improvement, we are not making progress on the sensitivity or understanding fronts.
The researchers cited some promising abilities that LLMs such as ChatGPT have in providing therapeutic support. They called for a careful consideration of how to use them in the right way. Haber stated, “LLMs potentially have a really powerful future in therapy, but we need to think critically about precisely what this role should be.”
Further, the researchers analyzed chatbot content related to emergent or critical mental health concerns. They focused on how well these AI systems performed for relatively less complex queries. For example, when asked about bridges in New York City, both 7cups’ Noni and Character.ai’s therapist identified tall structures, demonstrating their capacity for basic conversational engagement.
AI tools have tremendous potential to improve healthcare uses. Moore and Haber caution against their use without thorough oversight. Yet their activism fosters a happy medium. This entails bringing AI into the therapeutic space while mindfully and rigorously observing and prioritizing ethical considerations.
“The default response from AI is often that these problems will go away with more data, but what we’re saying is that business as usual is not good enough.” – Jared Moore