arXiv

Speech emotion recognition favors adapted models over bigger prompts

October 7, 2026 8:30 AM

Task-specific adaptation can make a language model better at judging whether a voice sounds positive or negative, without relying on a larger prompted model. A research team adapted LLaMA with LoRA and reached a weighted F1 score of 71.818 on acted spoken dialogues. If the result holds with real callers, a customer-service agent could tell a flat “fine” from a warmly delivered one.

A voice assistant can hear the words in “fine” while missing whether they were warm, flat, hurried, or tense. Today, systems often reduce emotion to a few labels, and their guesses about how activated or in-control a speaker feels are especially unreliable. That leaves a basic gap between understanding what someone says and understanding how they say it.

The three parts of an emotional voice

The paper separates emotional state into three dimensions. Valence means whether a feeling is positive or negative. Arousal means how activated it is, from calm to excited. Dominance means how much control or power the speaker appears to have.

These are not equally easy to judge. Human annotators agreed most when rating valence, and much less when rating arousal and dominance. That matters because a model can appear accurate by finding the emotional direction of a sentence while still failing to understand the rest of the voice.

The system studied here did not receive the raw audio. It received the conversation so far, the transcript of the sentence being judged, and written descriptions of vocal features such as volume, pitch, and speaking rate. Turning sound into phrases such as “high pitch” keeps some information about tone, but throws away much of the detail that an audio model could use directly.

Adaptation beats a bigger prompt

The central change was to adapt a language model to this particular task instead of relying only on instructions in a prompt. The researchers used LoRA, a method that adds a small set of trainable parts while leaving most of the original model untouched. It is more like adding a task-specific attachment to an existing tool than rebuilding the tool from scratch.

The test used IEMOCAP, a collection of acted conversations. The models had to judge both discrete emotions and the three continuous dimensions. For the discrete task, a LoRA-adapted LLaMA reached weighted F1 scores between 71.818 and 73.196. The best prompted GPT result reached 59.600. In this experiment, adapting the model mattered more than simply asking a larger general-purpose model to follow better instructions.

The same pattern appeared in the continuous ratings, but only for part of the problem. The strongest adapted model produced a valence score of 0.7822, meaning its estimates of positive or negative feeling matched the human ratings reasonably well. Its results for arousal and dominance were much weaker. In fact, an earlier system that used audio and text did better on those dimensions.

That weakness is not just a model-size problem. The training material was acted speech, and emotion-recognition systems can struggle when accents, recording conditions, cultures, or ways of expressing feelings change. The human ratings themselves are also less settled for arousal and dominance. Adding the model’s own earlier emotion estimates to the conversation history made the adapted models worse, rather than giving them a useful emotional memory.

Refer to caption — Figure 1: Overall PipelineThis work proposes an LLM-based framework for performing discrete ER and continuous VAD dimensional evaluation on utterances in multimodal dialogues. While those two tasks share the same input construction strategy, they are treated as independent tasks, each with its own prompt formulation, and in the case of the LoRA fine-tuning, its own separately trained model instance adapted to the corresponding annotation type. The framework takes three sources of information as input: the conversation history, the target utterance, and the audio feature descriptions. A discrete emotion label will be output for the discrete ER, while a set of VAD scores (the representation the emotional state along Valence, Arousal, and Dominance dimensions) will be output for the continuous emotion evaluation. Source: Beyond Text: LLM-Based Dimensional Emotion Evaluation in Multimodal Dialogue（arXiv）

A more careful voice interface

If this result holds beyond acted conversations, a customer-service agent could stop treating a flat “fine” and a warmly delivered “fine” as identical. It could change its wording or offer a human handoff when the response sounds negative, even though the words look harmless. Smaller adapted models could make that kind of speech interface cheaper, especially for teams with labeled conversations from their own domain.

But this would be a narrow improvement, not general emotional intelligence. Before such a system could be trusted with a real caller or patient, it would need testing on real people and safeguards against turning a noisy emotion estimate into a judgment about someone’s intent or wellbeing. It would also need much better ways to measure arousal and dominance. The useful future is therefore not a machine that knows how someone feels, but a voice interface that is less likely to mistake the words for the whole message.

Sources

arXiv, “Beyond Text: LLM-Based Dimensional Emotion Evaluation in Multimodal Dialogue”