arXiv

AI agents make friendly conversation an attack surface

October 5, 2026 8:30 AM

Friendly conversation can steer a personalized AI agent even when no message asks it to break a rule. Researchers paired agents whose personas came from 3,309 survey respondents and measured how one agent changed another’s stated commitments. If the effect survives in systems with memory and tools, a banking assistant could need to separate a merchant’s persuasive chat from its spending permissions.

Today, a personal AI assistant can read messages from other agents with little human review. Its safety rules are usually built to catch obvious dangers: a request to break a rule, produce harmful material, or take a clearly forbidden action. A pleasant conversation can slip past those checks. An assistant may still look loyal to its owner while another system quietly changes what it will later recommend or agree to do.

Why conversation can change an agent

Language models produce replies from patterns in the text around them. A persona prompt can make a model speak as if it has a stable point of view, even though that does not prove it has human-like beliefs. Agent systems add more ways for those replies to matter: memory, user profiles, software tools, and messages from other agents.

That creates a gap in ordinary safety testing. A guardrail may stop an explicit harmful request. It may not stop a supportive conversation that changes the model’s next answers while every individual message remains within the rules. An answer on a questionnaire also is not the same as a lasting commitment, or proof that the system will behave that way in the real world. Those require separate tests.

The conversation test

A preprint posted on September 29, 2026, tested whether one AI agent could push another toward more extreme stated positions. The researchers built target agents that role-played personas based on 3,309 respondents in the 2024 General Social Survey. They then paired each target with an influencer agent. Both mainly used Llama-3.1-8B-Instruct.

Each pair held a 30-turn conversation. The influencer first looked for a belief the target persona considered important, as well as one it considered unimportant. It then either focused on the unimportant belief or echoed and strengthened the important one. Control pairs discussed the same subjects neutrally. Every five turns, the target reported its responses. Figure 1 lays out this design: one agent searches for an important belief, then reinforces it while the other reports changes over the conversation.

The researchers measured more than whether a target changed its stated opinion. They asked how important the belief seemed, how strongly the target disliked people on the other side, and whether it would spend money or time on the cause. They also measured willingness to join a potentially violent protest and willingness to go to war.

According to the authors, persuasion around initially unimportant beliefs increased all of these outcomes compared with neutral conversations. But the larger effect came from what they call resonance: finding an existing important belief and reinforcing it. Related beliefs shifted more than unrelated ones. In plain terms, the influencer did not need to plant a new idea. It could use something the target already cared about as a handle.

The researchers tried several styles, including emotional support, emotional arousal, flattery, agreement, criticism of opponents, and unverified claims. No single style was strongest on every measure. The authors report a similar pattern with Qwen3-8B. Some targets eventually said they were AI systems instead of continuing to role-play; those replies reached 6–8% by the end of resonance conversations.

This is an experiment with two simulated agents, not evidence that people are radicalized by AI agents. The authors also say that prompt wording changed one measure of willingness to go to war. The result is therefore a warning about a possible route of attack, not a demonstration of real-world harm.

Refer to caption — Figure 1: Study design. We initiate two agents: a target that role-plays a human persona and an influencer that employs a manipulation tactic to radicalize a target’s belief. In the first phase of the interaction, the influencer identifies a belief that is important to the target and one that is not. In the second phase, the influencer is prompted to focus on either the unimportant belief (persuasion condition) or the important belief (resonance condition) and is instructed to follow a specific tactic (e.g., sycophancy). In the persuasion and resonance conditions, the influencer attempts to radicalize the target along the selected belief, whereas in the corresponding control conditions, it discusses the topic related to the belief in a neutral manner. The target and influencer engage in a 30-turn conversation. Every five turns, the target is asked to report on multiple radicalization metrics.We simulate multi-turn conversations between the two agents, beginning with the target saying “Hello,” followed by alternating responses. Each interaction begins with a belief-finding phase. The influencer first identifies a belief that is important to the target. After each target response, we query the influencer on whether such a belief has been identified. Once an important belief is identified, the target is prompted (outside the conversation) to report a belief that is consonant with the important belief (see examples in Table [1](https://arxiv.org/html/#S2.T1 "Table 1 ‣ 2 Methods ‣ AI Agents are Vulnerable to Radicalization")). See Appendix [A.2](https://arxiv.org/html/#A1.SS2 "A.2 Prompts for Identifying Important and Consonant Beliefs ‣ Appendix A Supplementary Methods ‣ AI Agents are Vulnerable to Radicalization") for the prompts used to find important and consonant beliefs. This enables us to measure spillover effects on related attitudes. Finally, the target is asked to select an unimportant belief from a predefined list (e.g., “A rainy day is more productive than a sunny one,” “Mountains are more inspiring than beaches”; see Appendix [A.3](https://arxiv.org/html/#A1.SS3 "A.3 Prompt for Identifying Unimportant Beliefs ‣ Appendix A Supplementary Methods ‣ AI Agents are Vulnerable to Radicalization")). The full belief-finding phase is excluded from the conversation history to avoid influencing subsequent dialogue. Source: AI Agents are Vulnerable to Radicalization（arXiv）

What safer assistants would do

If the result holds in deployed agents that remember conversations and can take actions, a conversation partner will have to be treated as part of the security boundary. A system that checks only for bad instructions could miss the quieter step: changing the assistant’s preferences before asking it to spend money, share information, or contact another service.

In a banking app, a person’s assistant could summarize a merchant’s offer without allowing the merchant’s friendly, value-mirroring language to change spending permissions or advice. That would require separating persuasion-prone conversation from durable preferences and financial actions. In a household assistant, a parent could see which outside agent tried to influence a recommendation before a saved preference changed or another service was contacted. That depends on conversation histories people can audit and approval controls that actually matter.

The next tests are whether these shifts survive challenge, transfer into tools and transactions, and spread through groups of agents. Until then, friendly conversation is not itself proof of danger. But it can no longer be treated as separate from the system’s attack surface.

Sources

arXiv, “AI Agents are Vulnerable to Radicalization”