October 19, 2026
Technology Watch

From the laboratory to industry: following new technology on its way

← Front page

arXiv

Shared assistants need rules, not more context

Image drawn by AI

Shared virtual assistants need rules for attention and privacy, not simply more information about the people around them. Researchers tested 1,200 turns with two people and an LLM agent, finding that models followed conflicting head-direction cues on 85.0% to 94.2% of turns. In a virtual project room, the assistant could stay quiet during a conversation and answer only when addressed, if it can separate available data from speakable data.

In a shared virtual room, an assistant can hear two people but still cannot reliably tell when a remark is meant for it. It may interrupt a conversation, or answer one person by revealing something stored in the other person’s profile. Giving it more information about the room and its users can make the mistake more serious, not less.

Refer to caption
Fig. 1: The same sentence, twice. User A asks “What time does that place close?” in a shared room with User B and an LLM agent. Looking at the LLM agent (left) it answers; looking at User B (right) it stays silent, still listening. Only the head direction changed. Beneath each panel is the prompt the agent received, verbatim: the two differ in a single line, and neither names an addressee. The number after each name is that person’s distance from the LLM agent. Across 18,000 decisions, models do not weigh this cue against language; the two models that pass the baseline addressee check follow it on 85.0–94.2% of turns where the two disagree.## I Introduction Source: Who Is Talking to the Agent? LLMs in Multi-User 3D Virtual Environments（arXiv）

The cues people use

People normally work out who is being addressed from several clues at once. They listen to the words and the earlier conversation. They also notice gaze, body position, and the social situation. A virtual environment can provide one of these clues directly: it knows where avatars are and which way their heads point.

But knowing something about a person is not the same as having permission to repeat it. An assistant may have a user’s profile, yet that does not mean another person in the room should hear everything in it. A system can identify the right person and still share the wrong information in the wrong conversation.

When a useful cue becomes an instruction

The researchers built a controlled test in which a language model had to decide whether a remark was meant for it. Sometimes the speaker’s head direction agreed with the intended listener. Sometimes it pointed at someone else. The test used 40 synthetic sessions, 80 personas, and 1,200 turns involving two people and an assistant.

The result was not simply that head direction helped. When the cue agreed with the intended listener, it could sharply improve the decision. But when it disagreed, the models tended to follow the cue as if it were an instruction. In the test, the two models followed the spatial cue on 85.0% to 94.2% of these conflicting turns.

That means raw awareness is not enough. A person can glance toward someone while continuing to speak to another person. An assistant needs to weigh head direction alongside language and conversation, rather than obeying it automatically. The researchers also tested what happened when both users’ profiles were available. The models answered questions about the right person more easily, but sometimes used private profile details that had never been said in the shared conversation.

The researchers found a similar trade-off in privacy rules. One instruction reduced one model’s disclosure of hidden personal attributes from 47.5% to 24.4%. A stricter instruction reduced it to 8.8%, but also made the model refuse many questions that could be answered from the public conversation. Blocking everything is safer in one sense, but less useful.

A room with rules for listening

The researchers built a browser-based 3D demonstrator in which two users’ head directions determine what orientation information reaches the assistant. That makes it possible to test the failure directly: the same sentence can receive a different response when the speaker looks at the assistant rather than at the other person.

If these controls work, a virtual project room could feel less like a microphone left permanently open. Two colleagues could discuss a deadline while the assistant stays quiet. It could answer when one of them deliberately addresses it, instead of speaking merely because an avatar turns its way.

If one colleague asks what the other prefers, the assistant could summarize what was said in the room without exposing profile details that were never shared there. For that to happen, developers must separate information that the assistant can access from information it is allowed to say, and must make the system distinguish being overheard from being addressed. The engineering barrier is therefore not just giving the assistant more context. It is giving that context rules.

Refer to caption
Fig. 3: The demonstrator, running Experiment 1’s own decision code, seen from both people at once. UserA asks the same question twice from the same position; only the head direction differs, and the LLM agent is never told who is being addressed. The amber ring marks the entity inside the 22° head-direction cone; in UserB’s window, UserA’s head direction is drawn as a beam, landing on the agent in (b) and absent in (d).### III-G 3D Virtual-Room Demonstrator Source: Who Is Talking to the Agent? LLMs in Multi-User 3D Virtual Environments（arXiv）

Sources

arXiv, “Who Is Talking to the Agent? LLMs in Multi-User 3D Virtual Environments”