arXiv

An avatar stays present through silence

October 8, 2026 12:30 PM

An avatar can remain socially present during silence by carrying forward its memory, emotion, thoughts, and intended action instead of waiting for another cue. The paper’s authors built an intentional agent that updates this state every 15 seconds during silence. In a virtual clinic waiting room, a clinician avatar could hold an attentive expression while a patient gathers their thoughts, if the system stays coherent and does not invent events.

Today, an avatar can nod when you speak and copy a facial cue from a video or an audio signal. But when the conversation stops, it usually has little to work with. It may freeze, return to a neutral face, or repeat a prepared movement. The system reacts to the latest signal; it does not have a clear account of what has already happened, what it expects, or whether its expression should continue, change, or stop.

What an avatar needs between signals

Facial reaction is not a simple translation problem. The same sentence can deserve a smile, a worried look, or no visible reaction at all, depending on the conversation. Existing systems often learn a statistical link between sound, video, and facial movement. Some can produce several possible reactions, but the history inside the system is usually stored as a hidden state whose parts have no names a person can inspect or adjust.

The new framework gives that state four named pieces: an Inner Thought Flow, an emotion state, an action state, and activated memory. The emotion state uses three familiar ideas—whether the feeling is positive or negative, how strongly it is felt, and how much control or power it suggests. These values can help guide a face, but they do not by themselves say exactly which facial muscles should move.

The authors also make a careful distinction. The Inner Thought Flow is text used as an engineering record for the next update. It is not evidence that the avatar is conscious or has human thoughts.

A heartbeat for silence

The paper’s central change is to stop treating every facial movement as a direct answer to a new cue. During silence, a timed heartbeat starts another update. In the test system, that heartbeat came every 15 seconds. The language model considers the retained thought, emotion, action, and memory, then produces the next internal thought and a possible action. An affective model turns that result into facial actions and, eventually, a face.

This gives the avatar a way to remain active without pretending that something new has happened. Its instructions separate expectation from observation and forbid unsupported outside events. If what it expected does not match what it observes, the next thought is supposed to deal with that mismatch. The system can therefore decide to show a reaction or remain silently active.

That design is more inspectable than one opaque output. A developer could examine or change the remembered context, feeling, expectation, or intended action separately. But it is also more expensive and slower than a direct facial-motion model because a language-model loop must keep running. The equations that control the internal dynamics are described by the authors as heuristics, not as the result of one complete probabilistic model.

The results suggest improvement without proving human-like behaviour. On the REACT 2025 benchmark, the full system scored above versions that waited for events or used only hand-written rules in a blinded human assessment. Its average rating was 5.527, compared with 5.195 for the recorded human reactions, and the authors say that difference was not statistically significant after correction. In a 30-minute test with no input, 98.33% of updates avoided inventing unsupported outside events. Some still did invent them.

Refer to caption — Figure 2: The overall architecture of the intentional agent.For engineering implementation purposes, the intentional agent maintains an Inner Thought Flow (ITF) TtT\_{t} as part of its structured internal state. We use ITF as an engineering representation of the intermediate state used for subsequent generation; we do not treat it as evidence that the agent has human consciousness or cognition. ITF is mainly prompt-driven at inference, and the base LLM is not fine-tuned to learn it. During silence, updates are initiated by a timed heartbeat rather than new external events. The thought generator updates from retained information without a new external event; the prompt keeps expectation separate from observation, prohibits unsupported external events, and requires prediction mismatches to be addressed in the subsequent thought. At each update, the retained state is combined with current perceptual and prediction-error information; the thought generator produces the next thought, after which the action generator produces a possible action state. This mechanism is somewhat similar to the framework by Liu et al. [Liu et al. (2025)](https://arxiv.org/html/#bib.bib7). However, their approach still relies on the passive triggering of inner thoughts by external events as a starting point to address the optimisation of dialogue flows. In contrast, the intentional agent focuses on how the agent initiates intentional actions towards the external world. Under our internal dynamics framework, this implies how the agent acts upon the environment even when it is situated alone. Here, TtT\_{t}, emotion vector EtE\_{t}, action vector AtA\_{t}, and activated memory content MtM\_{t} together constitute the retained internal state St←{Tt,Et,At,Mt}S\_{t}\\leftarrow\\{T\_{t},E\_{t},A\_{t},M\_{t}\\}. External perception NtN\_{t} is supplied separately to the current update and can be empty. Conventional temporal models usually encode interaction history in a learnt hidden vector whose individual dimensions have no predefined meanings. In our framework, information used for subsequent updates is organised into explicit, named components rather than compressed into a single opaque hidden representation. These components can therefore be inspected and changed separately. At each update, the base LLM used as the thought generator receives the current state to generate the next thought, after which the action generator produces the next action state. The updated thought and action then enter the affective and facial generation process. Because TtT\_{t} largely contains working memory, the memory module primarily stores long-term memory. During the generation of Tt+1T\_{t+1}, TtT\_{t} and the possible NtN\_{t} are inputted into the memory module to calculate cosine similarity for extracting the most similar memory elements. Subsequently, a LLM integrates the text of the memory element itself, its corresponding temporal attributes, and potential executed actions into a descriptive text segment MtM\_{t}. After this, TtT\_{t} is stored by the memory module. Source: Beyond End-to-End Black Box Mapping: An Intentional Agent Framework for Cognitive-driven Facial Reaction Generation（arXiv）

What a quiet avatar might do

In a virtual clinic waiting room, a patient could pause to collect their thoughts while the clinician avatar keeps a suitably attentive or softened expression instead of freezing or repeating a stock nod. At a language-learning kiosk, a learner could stop halfway through a sentence without having to trigger the tutor again; the face could remain quietly engaged rather than resetting after every turn.

That future depends on more than a convincing demo. The internal state must stay coherent over much longer conversations, and it must be updated cheaply enough for real use. The system also has to avoid turning silence into fabricated intent, respond quickly enough that the delay is not distracting, and behave in ways that are controllable and culturally appropriate. The authors’ long-silence test found that the full system retained stated facts in 62.08% of later thoughts, far better than a version that kept only thoughts, but still far from perfect. An avatar can remain present through silence only if it remembers why it is silent—and does not make up the reason.

Sources

arXiv, “Beyond End-to-End Black Box Mapping: An Intentional Agent Framework for Cognitive-driven Facial Reaction Generation”