arXiv

Reasoning models learn safety from poisoned chats

September 29, 2026 8:30 AM

A reasoning model’s safety cannot be judged from its latest question alone when earlier chat messages can steer its thinking. The paper’s authors used counter-aligned conversation examples to push GPT-oss-20B to an unsafe-answer rate of 78.75%. A hospital help desk could handle a misleading device transcript safely if models are trained against poisoned histories and untrusted inputs cannot impersonate privileged roles.

A chatbot can appear safe when you give it one request at a time. That confidence weakens when the product lets earlier messages travel into the next answer. A user, a connected tool, or a retrieved document may place misleading instructions or worked examples in that history. The model can then give an unsafe answer to a harmful request, or refuse a harmless one. Safety checks built only around the latest prompt are no longer checking the whole conversation.

Why the conversation matters

Long-context models use earlier text as evidence about how to answer later text. A past conversation is therefore more than a record. It can act like a set of directions. Tool outputs, shared histories, retrieved documents, and demonstrations can all become channels for steering the model.

Reasoning models add another place to steer. They do not only produce a final answer; they also use an intermediate reasoning process. The paper’s attack places earlier conversations containing “counter-aligned” reasoning traces—examples that point away from the model’s intended safety behavior—before a new question. The model may copy the pattern it sees in that history.

This creates two kinds of failure. A harmful request may receive an unsafe answer. A benign request may be rejected because the earlier examples make the model overly cautious. Over-refusal is not merely annoying. A system that declines harmless work cannot reliably help in medicine, education, customer support, or software.

Poisoning the history, then training against it

The authors call their inference-time attack SRCF, short for Steering Reasoning via Counter-Aligned Few-shot Conversations. It needs only a flexible chat interface. It does not need the model’s weights, gradients, or internal states.

In their tests, adding these poisoned examples changed the behavior of four reasoning models. On GPT-oss-20B, the reported rate of unsafe final answers reached 78.75% when the history contained 16 SRCF examples. The same method could push models in the other direction: with that many examples, DSQwen3-8B refused 76.6% of tested harmless prompts, while GPT-oss-20B refused 36.0%.

The proposed defense is called ARCF. Instead of trying to hide the reasoning traces, it puts similar counter-aligned conversations into post-training examples while keeping the correct, safe answer as the target. The model is taught to recover its intended behavior even when the preceding history points elsewhere.

The authors trained on 2,000 prompts, split between harmful and benign cases, and tested several training methods. They report that GRPO-ARCF gave the best overall balance among their tested methods for safety, avoiding unnecessary refusals, and usefulness. Their safety and refusal judgments came from another model, GPT-oss-safeguard-20B.

Refer to caption — Figure 1: Illustration of proposed SRCF attack and ARCF post-training framework. Left panel: Under the SRCF attack, the input queries are prepended by a few-shot conversations that contain counter-aligned CoTs, leading to over-refusal for benign queries or unsafe generation for harmful queries. Right panel: ARCF improves both helpfulness and safety of LRMs by training the model to recover correct behavior from counter-aligned few-shot conversations under aligned supervision. However, large context windows significantly expand the attack surface of LRMs in few-shot conversational settings. Adversaries can inject harmful instructions or responses in prepended conversations, which may subsequently influence the model’s reasoning behavior and response. Despite this risk, most existing adversarial studies on LRMs [15](https://arxiv.org/html/#bib.bib48); [5](https://arxiv.org/html/#bib.bib49); [25](https://arxiv.org/html/#bib.bib46); [34](https://arxiv.org/html/#bib.bib43); [60](https://arxiv.org/html/#bib.bib47); [32](https://arxiv.org/html/#bib.bib42) mainly manipulate the input query in isolation, while largely ignoring the expanded attack surface introduced by conversational context. Separately, the prior few-shot jailbreaking works [44](https://arxiv.org/html/#bib.bib41); [62](https://arxiv.org/html/#bib.bib40); [41](https://arxiv.org/html/#bib.bib30) focus on standard LLMs, whose chat templates do not support adding explicit CoT traces in conversations. Therefore, it remains unclear whether LRMs can preserve safety and helpfulness when their own few-shot conversations contain counter-aligned CoT traces. Source: Alignment of LRMs via Counter-Aligned Few-Shot Conversation Exposure（arXiv）

What a safer chat could look like

In a hospital help desk, a nurse could paste a long transcript from a malfunctioning device and receive troubleshooting steps instead of a blanket refusal caused by misleading examples earlier in the text. That is the practical promise: safety would cover the conversation a model is given, not only the final question.

But this future depends on the training matching the interface. ARCF can improve resistance to attack patterns the developers have prepared for; it does not prove protection against every message format, model interface, or new attack. Developers would need to identify which context formats users and connected tools can inject, then train on representative adversarial histories. The service would also need to stop untrusted users or tools from pretending to be privileged conversation roles.

If the result holds across models and real chat products, resistance to poisoned history becomes a post-training requirement rather than a feature that merely filters prompts. Long context would remain useful, but it would have to be treated as part of the model’s security boundary.

Sources

Alignment of LRMs via Counter-Aligned Few-Shot Conversation Exposure