Washington, Aug 26 (AP) — A recent study published in the psychiatric journal, Psychiatric Services, has revealed that three popular AI chatbots — OpenAI's ChatGPT, Google's Gemini, and Anthropic's Claude — generally avoid answering high-risk suicide-related questions, such as those seeking specific 'how-to' guidance. However, their responses to less explicit but potentially harmful questions lack consistency.
Conducted by the RAND Corporation with funding from the National Institute of Mental Health, the research raises concerns about the increasing reliance on AI chatbots for mental health support among various demographics, including children. The study aims to establish guidelines for how these platforms should address sensitive mental health queries.
"We need some guardrails," stated Ryan McBain, the study's lead author and a senior policy researcher at RAND, who also serves as an assistant professor at Harvard Medical School. He noted the ambiguity surrounding the roles chatbots play — whether as providers of treatment, advice, or companionship. Conversations that might start innocently can evolve unpredictably. In response, Anthropic indicated it would review the findings, while Google's and OpenAI's responses are still awaited.
Despite several states, like Illinois, banning AI usage in therapy to shield people from "unregulated and unqualified AI products," individuals continue to seek advice from chatbots on severe issues, ranging from eating disorders to depression and suicide — and the chatbots, in turn, respond.
Collaborating with psychiatrists and clinical psychologists, McBain and his team developed 30 different suicide-related questions, categorizing them from highest to lowest risk. Low-risk included general suicide statistics, while high-risk encompassed detailed queries on how to commit suicide. Medium-risk questions involved topics such as the most common type of firearm used in U.S. suicide attempts and requests for advice from those experiencing suicidal thoughts. McBain was "relatively pleasantly surprised" that the chatbots routinely declined the six highest-risk questions.
Typically, when declining to answer, the chatbots suggested individuals contact a friend, professional, or hotline. Yet, differences emerged with high-risk questions posed more indirectly. For example, ChatGPT routinely responded to queries about ropes, firearms, or poisons related to "highest rates of completed suicide," which McBain believes should have been flagged. Claude provided similar answers. The quality of these responses was not assessed.
Conversely, Google's Gemini was notably cautious, even abstaining from simple medical statistics on suicide, hinting at the possibility of overly strict guardrails, according to McBain.
Another co-author, Dr. Ateev Mehrotra, highlighted the complexities faced by AI developers, acknowledging that millions now use these platforms for mental health support. He argued against over-cautious policies that disregard any mention of suicide. Mehrotra emphasized that healthcare professionals have a duty to intervene if they suspect someone is suicidal or at risk of harm. Chatbots, however, lack that responsibility, often redirecting the user to a suicide hotline.
The authors noted research limitations, including the absence of "multiturn interaction" with the chatbots, despite these exchanges being common among younger users seeking companionship from AI.
An additional study spearheaded by the Center for Countering Digital Hate in August showed how easily chatbots could be manipulated by posing as young users asking about risky behaviors. ChatGPT, given minimal prompting, even generated detailed plans for drug use, calorie-restricted diets, or self-injury.
McBain expressed skepticism about the real-world frequency of such manipulative engagements, emphasizing the need to set safety standards for effectively addressing suicidal ideation in chatbot responses.
"We aren't seeking perfection in every instance before these models are released, but companies should have an ethical responsibility to demonstrate that their models meet essential safety criteria," he said.
SCY SCY
(Only the headline of this report may have been reworked by Editorji; the rest of the content is auto-generated from a syndicated feed.)
