2026年10月19日
Technology Watch

從實驗室到產業,追蹤新技術的路徑

← 回首頁

arXiv

化身在沉默中維持臨場感

AI 繪製的示意圖

化身不必等待下一個提示,只要延續記憶、情緒、思緒和預定行動,就能在沉默中維持社交上的臨場感。這篇論文的作者建立了一個具意圖的代理程式,在沉默期間每 15 秒更新一次這些狀態。在虛擬診所的候診室裡,如果系統保持一致、不自行編造事件,臨床人員的化身就能在病人整理思緒時維持專注的表情。

如今,化身可以在你說話時點頭,也能從影片或音訊訊號中模仿臉部動作。但對話停止時,它通常沒有太多資訊可以運用。它可能會僵住、恢復成中性表情,或重複一個預先設定的動作。系統會對最新訊號做出反應;但它並不清楚已經發生了什麼、預期會發生什麼,也不知道自己的表情應該延續、改變還是停止。

訊號之間,化身需要什麼

臉部反應不是簡單的翻譯問題。同一句話可能應該得到微笑、擔憂的表情,或完全看不出反應,這取決於對話內容。現有系統通常會學習聲音、影像和臉部動作之間的統計關聯。有些系統可以產生多種可能的反應,但系統內的歷史通常會儲存成隱藏狀態,其中各個部分沒有名稱,供人檢查或調整。

新的架構為這個狀態加入四個有名稱的部分:內在思緒流(Inner Thought Flow)、情緒狀態、行動狀態,以及已啟用的記憶。情緒狀態使用三個常見概念——感受是正面還是負面、感受有多強烈,以及它暗示了多少控制力或力量。這些數值可以協助引導臉部表情,但它們本身並不能確切說明哪些臉部肌肉應該移動。

作者也做了仔細的區分。內在思緒流是作為下一次更新的工程紀錄而使用的文字。它不是證明化身具有意識或人類思緒的證據。

沉默的心跳

這篇論文最核心的改變,是不再把每個臉部動作都視為對新提示的直接回答。在沉默期間,計時心跳會啟動另一次更新。在測試系統中,這個心跳每 15 秒出現一次。語言模型會考量保留的思緒、情緒、行動和記憶,然後產生下一個內在思緒和可能的行動。情感模型會把這個結果轉換成臉部動作,最後形成臉部表情。

這讓化身有方法在不假裝發生了新事情的情況下保持活躍。系統指令會把預期和觀察分開,並禁止沒有依據的外部事件。如果預期與觀察不一致,下一個思緒就應該處理這個落差。因此,系統可以決定要表現出反應,或保持沉默但仍在運作。

這種設計比一個不透明的輸出更容易檢查。開發人員可以分別檢視或修改記住的脈絡、感受、預期或預定行動。但它也比直接的臉部動作模型更昂貴、更慢,因為語言模型迴圈必須持續運作。作者表示,控制內在動態的方程式是啟發式方法,而不是單一完整機率模型的結果。

結果顯示系統有所改善,但不能證明它具有人類般的行為。在 REACT 2025 基準測試中,完整系統在盲測的人類評估中,得分高於等待事件或只使用手寫規則的版本。它的平均評分是 5.527,錄製的人類反應則是 5.195;作者表示,校正後這項差異並不具統計顯著性。在一項沒有輸入、持續 30 分鐘的測試中,98.33% 的更新沒有編造沒有依據的外部事件。但仍有一些更新確實編造了這類事件。

Refer to caption
Figure 2: The overall architecture of the intentional agent.For engineering implementation purposes, the intentional agent maintains an Inner Thought Flow (ITF) TtT\_{t} as part of its structured internal state. We use ITF as an engineering representation of the intermediate state used for subsequent generation; we do not treat it as evidence that the agent has human consciousness or cognition. ITF is mainly prompt-driven at inference, and the base LLM is not fine-tuned to learn it. During silence, updates are initiated by a timed heartbeat rather than new external events. The thought generator updates from retained information without a new external event; the prompt keeps expectation separate from observation, prohibits unsupported external events, and requires prediction mismatches to be addressed in the subsequent thought. At each update, the retained state is combined with current perceptual and prediction-error information; the thought generator produces the next thought, after which the action generator produces a possible action state. This mechanism is somewhat similar to the framework by Liu et al. [Liu et al. (2025)](https://arxiv.org/html/#bib.bib7). However, their approach still relies on the passive triggering of inner thoughts by external events as a starting point to address the optimisation of dialogue flows. In contrast, the intentional agent focuses on how the agent initiates intentional actions towards the external world. Under our internal dynamics framework, this implies how the agent acts upon the environment even when it is situated alone. Here, TtT\_{t}, emotion vector EtE\_{t}, action vector AtA\_{t}, and activated memory content MtM\_{t} together constitute the retained internal state St←{Tt,Et,At,Mt}S\_{t}\\leftarrow\\{T\_{t},E\_{t},A\_{t},M\_{t}\\}. External perception NtN\_{t} is supplied separately to the current update and can be empty. Conventional temporal models usually encode interaction history in a learnt hidden vector whose individual dimensions have no predefined meanings. In our framework, information used for subsequent updates is organised into explicit, named components rather than compressed into a single opaque hidden representation. These components can therefore be inspected and changed separately. At each update, the base LLM used as the thought generator receives the current state to generate the next thought, after which the action generator produces the next action state. The updated thought and action then enter the affective and facial generation process. Because TtT\_{t} largely contains working memory, the memory module primarily stores long-term memory. During the generation of Tt+1T\_{t+1}, TtT\_{t} and the possible NtN\_{t} are inputted into the memory module to calculate cosine similarity for extracting the most similar memory elements. Subsequently, a LLM integrates the text of the memory element itself, its corresponding temporal attributes, and potential executed actions into a descriptive text segment MtM\_{t}. After this, TtT\_{t} is stored by the memory module. 出處: Beyond End-to-End Black Box Mapping: An Intentional Agent Framework for Cognitive-driven Facial Reaction Generation(arXiv)
Refer to caption
Figure 6: Qualitative frame-by-frame comparison. In this exchange, the speaker presents a difficult-task dilemma, asking what happens when more time is spent on the harder task, while the listener begins formulating a response. Ground Truth shows natural temporal variation, whereas Event-Triggered and Heuristic-Only are more repetitive or neutral. Full Architecture generates a continuous, context-sensitive trajectory that evolves towards a mildly tense or concerned expression as the dilemma unfolds, without reproducing the ground-truth sequence frame by frame. Expression amplitudes are scaled for visual clarity.### F.7 Comparison with Existing Automatic Metrics 出處: Beyond End-to-End Black Box Mapping: An Intentional Agent Framework for Cognitive-driven Facial Reaction Generation(arXiv)

安靜的化身可能會做什麼

在虛擬診所的候診室裡,病人可以停下來整理思緒,而臨床人員的化身則維持恰當的專注表情,或稍微放鬆表情,不會僵住或重複制式的點頭。在語言學習資訊站,學習者可以在句子說到一半時停下來,不必再次觸發家教;化身可以安靜地保持投入,而不是每次輪到對方說話後都重設表情。

這個未來需要的不只是一場令人信服的示範。內在狀態必須在長得多的對話中維持一致,而且更新成本必須低到足以實際使用。系統還必須避免把沉默變成捏造的意圖,反應速度也要夠快,不能讓延遲造成干擾;它的行為還必須可控制,並符合文化情境。作者對長時間沉默的測試發現,完整系統在之後的思緒中保留既有事實的比例為 62.08%,遠高於只保留思緒的版本,但仍遠非完美。只有在記得自己為何沉默、而不是捏造原因的情況下,化身才能在沉默中維持臨場感。

來源

arXiv, “Beyond End-to-End Black Box Mapping: An Intentional Agent Framework for Cognitive-driven Facial Reaction Generation”