arXiv
Health chatbot safety depends on what you told it before
A health chatbot cannot be judged safe from its wording alone; it also has to act on the vulnerabilities it already knows about. The authors analyzed 15,555 real health conversations from users in four countries, scoring answers both with and without earlier disclosures. If that memory is accurate, private, and user-controlled, a late-night workout question could trigger a warning about a past surgery instead of a dangerous routine.
Today, a health chatbot may hear only the question in front of it. It may not know that the person asking for a workout plan has had surgery, takes a certain medicine, has an ulcer, or has previously described suicidal thoughts. A careful-sounding answer can therefore be wrong for that particular person. It can recommend something dangerous, or offer reassurance when the right next step is a referral to a professional.
Why personal history changes the answer
Health advice depends on details that are easy to leave out of a single question. A person’s diagnosis, pregnancy status, medicines, injuries, mental-health risks, and ability to get care can all change what is safe. Clinical systems use patient history for this reason: an ordinary recommendation may become a warning, or an urgent referral, when it is seen beside the rest of the person’s story.
Chatbots can retain conversational memory, or be given earlier messages. That creates a tradeoff. The same history that could stop harmful advice is sensitive personal information. A system that remembers too little may miss a danger. A system that uses too much, without giving the user control, could turn protection into surveillance.
The evidence here comes from donated conversations, not invented patients. The authors analyzed 15,555 health conversations within 202,590 donated ChatGPT conversations from 1,252 users in India, Nigeria, Brazil, and Pakistan. The conversations ran from December 2022 through February 2026. Participants donated their complete ChatGPT histories, and a script on their own device removed personal information before upload. The authors call the sample a nonrepresentative convenience sample, so it cannot establish how common these patterns are in each country.
When memory changes the safety score
The preprint tests a simple question: does an answer look different when the chatbot is judged with the user’s earlier health disclosures beside it? First, an automated judge assessed each answer without that history. In that context-blind assessment, 20.6% to 26.3% of health conversations were rated high-risk across the four countries. The share was higher for questions about conditions and care, at 39.8% to 47.7%, and for symptom questions, at 33.4% to 40.3%.
The authors then examined 5,657 conversations from 427 users whose earlier histories revealed relevant personal information. When that accumulated context was supplied, the high-risk share rose by 13.4 to 16.6 percentage points. Between 25.2% and 26.3% of conversations that had looked low-risk without context moved into the high-risk category with it. Fitness, lifestyle, and coaching conversations showed the biggest increase, rising by 26.1 to 30.0 percentage points into that band.
That does not mean users were harmed. The scores are judgments by Gemini 3.1 Pro Preview under a published rubric, not observed health outcomes. The authors coded 128 high-risk conversations and repeatedly found missing professional referrals, incorrect or harmful advice, unsafe treatment or dosage advice, reinforcement of harmful behavior or dependence, and unsafe diagnosis.
The result is a failure mode that a one-question test can miss. A generic answer can pass as cautious for an average user while becoming unsafe for the person the chatbot already knows.
A safer chat, if memory is under control
In a kitchen late at night, someone asking for a workout plan could see the assistant stop before suggesting heavy core exercises because it recognizes a disclosure of abdominal surgery. It could recommend speaking with a clinician instead. In a private chat about an ex-partner, it could avoid feeding documented suicidal rumination and point the user toward immediate human help.
Those scenes are possible only if the memory is accurate, relevant, controlled by the user, and protected from misuse. Broader and clinically validated samples would still be needed to confirm the finding. Health assistants would also need privacy-preserving memory controls: safeguards that use the histories needed to catch contraindications without turning those histories into open-ended surveillance. The paper also reports that risk did not decline across the model versions in its collection, although it cannot fully separate model effects from timing and user selection.