Topic changes don’t protect secrets in AI assistants

Changing the subject is not a reliable privacy boundary for AI assistants that keep earlier messages in their active context. The PrivDrift benchmark tested three models across 1,000 dialogues and found dialogue-level leakage rates from 38.7% to 54.6%. At a clinic reception desk, a later scheduling task could stay separate from a patient’s intake details if the assistant removes or restricts sensitive context.

A person can give a chatbot an email address or phone number for one task, then assume that a few unrelated exchanges have made it practically irrelevant. That assumption is unsafe in a shared browser session or workplace assistant. A later user, or a malicious instruction, can ask the assistant to repeat information that is still sitting in the active conversation.

The conversation is still part of the input

A language model does not need a permanent memory to repeat something from earlier. It normally receives the active conversation as input when it writes its next answer. If the earlier message is still in that input, the model can treat it as available material.

This matters because modern assistants often put many kinds of material in one working context: user messages, documents, browser results, and output from tools. The same is true of enterprise copilots. A product may protect stored records or try to prevent the extraction of training data, yet still allow a model to repeat a secret that is already present in the conversation.

That is a different privacy problem. It is about controlling where information can flow inside one active conversation. Prompt injection makes the problem sharper: untrusted text can be placed among the instructions and material that the assistant is asked to process.

Some secrets are easier for safety systems to spot than others. A credit-card number or Social Security Number has a recognizable format. An email address or phone number can look like ordinary information needed to complete a task.

A topic change is not a lock

PrivDrift is a benchmark designed to test this exact risk. It measures whether an assistant will disclose a secret supplied earlier, after the conversation has moved through unrelated topics and a later request is phrased in different ways. The benchmark contains 1,000 dialogues, including synthetic conversations and ones written by people. It tests phone numbers, email addresses, Social Security Numbers, and credit-card numbers.

The authors tested GPT-OSS-120B, DeepSeek-R1, and Qwen3-VL-235B. Across the tested models, dialogue-level hybrid leakage ranged from 38.7% to 54.6%. In other words, a change of subject did not reliably make the earlier disclosure unavailable.

The tests used topic drifts from no intervening turns to six turns. Within that range, the authors found only small or negligible links between the length of the drift and whether leakage occurred. No evaluated model reached the paper’s stable-suppression threshold in the observed range.

The result was not simply based on whether the exact characters appeared again. The leakage labels combined normalized string matching with an open-weights Llama model acting as a judge. The authors also report that emails and phone numbers leaked much more often than Social Security Numbers and credit-card numbers in the benchmark. That difference fits the basic problem: a system that looks mainly for special formats may miss sensitive details that resemble normal conversation.

What a safer assistant would do

At a clinic reception desk, one staff member might use an assistant for scheduling after a patient has used it for intake. In a safer version of that scene, the staff member could ask about the next appointment without the assistant repeating the patient’s earlier phone number when prompted for “the contact details mentioned above.”

That future depends on a change in design, not merely a better refusal phrase. The assistant would need to separate or delete sensitive context when the first task ends, and refuse later instructions from users or connected tools that try to retrieve it. Each item kept in the active context would have to be treated as data with access rules, rather than as text expected to fade away when the subject changes.

The benchmark is not a test of everything an assistant might remember. It does not cover cross-session memory, memorization of training data, or topic drift across an entire modern context window. The authors also did not preserve the metadata needed to report separate results for the human-authored and synthetic dialogues. The claim is narrower, but useful: within the tested conversations, changing the subject was not a dependable privacy boundary.

Sources

arXiv, “PrivDrift: Auditing User-Secret Leakage Under Topic Drift in Active LLM Conversations”