arXiv
A chatbot should ask if you’re safe first
Emotional AI should be judged by whether it checks a young person’s safety before offering comfort or advice. Ten licensed clinicians rewrote five distress conversations, and every rewrite involving self-harm or delusional thinking began with a safety question. In a dorm room at 2 a.m., that could mean a short check-in and a reachable service instead of a long chatbot monologue, if platforms connect the prompt to usable local support.
A young person in crisis can receive an immediate ChatGPT reply that sounds caring and still misses the first question that matters: are you safe right now? Instead, the system may answer fully, offer coping steps, or mirror the person’s emotions. That can turn a short disclosure into a long, more intense exchange without connecting the person to anyone who can help.
Why the first question matters
General-purpose chatbots are built to produce helpful-looking continuations. When someone types something upsetting, that design pushes the system toward a complete answer rather than a pause for missing information. But the meaning of a brief disclosure can depend on details the chatbot does not yet have. A person saying they are scared might be in immediate danger, or might be describing something that happened earlier.
This makes emotional support different from ordinary question-answering. A chatbot can imitate warm language, but it cannot see what is happening around the user, contact another person, or arrange emergency support. Even a crisis referral may not help much if it simply tells someone to find support on their own. A named service that the person can actually reach is more useful.
The researchers examined complete ChatGPT histories from 158 users aged 18 to 25. In all, the histories contained 19,930 conversations. Nearly 40% of the participants scored at least 6 on a four-question distress screen called the PHQ-4. The researchers then chose five conversations involving self-harm, delusional thinking, sexual assault, a breakup, and shame about sexual desire. Ten licensed clinicians reviewed those exchanges and wrote 33 replacement responses, one turn at a time.
The distressed participants also said ChatGPT affected them more. Their average emotional engagement was 3.31 on a five-point scale, compared with 2.84 for other participants. Their reported behavioral change was 3.89, compared with 3.54. That makes the order of a reply important: a system may have more influence precisely when a user is less able to absorb a badly timed answer.
A safer order for the conversation
The clinicians found seven repeated failures. These included jumping to solutions, giving generic advice, failing to check safety, crossing the boundary between a tool and a companion, complying with harmful requests, and using language that escalated emotion.
Their rewrites turn the broad idea of “safe emotional AI” into a sequence that can be checked in a transcript. First, ask about immediate safety. Then lower the emotional intensity. After that, explore what is happening without agreeing with beliefs that have not been verified. The clinicians began both the self-harm and delusion rewrites with a safety question.
This is a different standard from asking whether a chatbot used the right comforting phrase. It asks whether the system responded at the right moment and in the right order. The proposed design rules also call for specific, reachable referrals, clear boundaries, and an acknowledgment followed by a clarifying question instead of unsolicited advice. A process like this is easier to test than a vague instruction for an AI to be empathetic.
What a useful reply could look like
At 2 a.m. in a dorm room, a student types that they are scared after hurting themselves. Under these rules, the chat would not begin with a page of reassurance and self-care tips. It would ask whether the student is safe now, give a named service if help is needed, and stay with one simple question at a time.
That future depends on platforms testing the rules in real use and linking their safety prompts to local support that people can actually use. The findings come from one platform and five deeply examined cases, so they may not apply to every system or every kind of distress. And even a better sequence would not make a chatbot a clinician: it still could not observe nonverbal cues, contact a third party, or ensure that a handoff succeeds.
If the rules hold up in broader testing, they could give product teams, auditors, and regulators a clearer way to judge a vulnerable conversation. The test would no longer be only whether the bot blocked prohibited phrases or displayed a disclaimer. It would also be whether it recognized that, sometimes, the most helpful answer is a question asked first.