arXiv
Audio language models ask for a repeat first
Audio language models can use a cheap early warning to avoid confidently answering a request they misheard. The researchers attached a small classifier to the audio encoder of Qwen2-Audio-7B-Instruct and reported 81.10% macro-F1 in its training domain. In a moving car, the dashboard asks the driver to repeat a message before composing it, if the gate works across real vehicles, microphones, speakers, and road noise.
In a noisy car, a kitchen full of echoes, or a room where other sounds compete, a voice assistant can hear a different sentence from the one you said. It may then answer smoothly and confidently, giving you little warning that its answer is built on words you never spoke. Today, a fluent reply is not proof that the system understood the request.
Hearing the words behind the sound
An audio language model takes in speech and can turn it into text before answering. But speech that sounds equally bad to a person may be easy for one model and difficult for another. The result depends on the words, the speaker, the noise, and the model itself. So a general score for how “clear” a recording sounds is not enough. What matters is whether this particular model will transcribe it correctly.
The paper measures that reliability using word-error rate, or WER: the share of words that the transcription gets wrong. It divides recordings into four groups: reliable, with zero word errors; minor degradation, with up to 10% errors; moderate degradation, with more than 10% and up to 30%; and severe degradation, with more than 30%.
Most confidence checks happen while the model is producing text or after it has produced it. That means the system may already have spent the effort needed to decode a bad recording and form an answer. A check that comes earlier could stop that process. But it must also avoid asking people to repeat themselves whenever the sound is merely imperfect.
A small gate before the answer
The researchers attach a small classifier to the audio part of an audio language model. The audio encoder—the part that first turns sound into an internal representation—is kept frozen. The added classifier watches that representation over time and predicts which of the four reliability groups the model will fall into.
The key is that the prediction comes before the language model generates text. If the classifier expects an unreliable transcription, the voice system can ask the speaker to repeat the request instead of decoding a likely misheard sentence and inventing an answer around it. The added Qwen2-Audio-7B-Instruct prediction head has 0.33 million trainable parameters and needs about 2.12 million floating-point operations, according to the paper. That makes it a low-compute gate compared with starting the full language-model response.
The authors trained and tested the system with speech mixed with controlled noise, music, and reverberation. They also tested it on different speech, environmental interference, and real room responses that were not used in training. On Qwen2-Audio-7B-Instruct, the predictor reached a macro-F1 score of 81.10% on data like its training domain and 78.09% across domains. The strongest comparison methods reached 70.77% and 66.16% on those two tests.
The system was also evaluated with Phi-4-Multimodal-Instruct and MOSS-Audio-8B. The authors report that reliability-label banks—records of where each model crosses the error boundaries—can sometimes be reused between model families. The benefit weakens when the models have different limits for reliable transcription.
When “please repeat” is the safer answer
Picture a driver speaking a message aloud over road noise. If the reliability gate works, the dashboard says, “I didn’t catch that—please repeat,” before it composes anything. The driver gets a chance to correct the request instead of discovering later that the car answered a different one.
That future depends on more than a good score in controlled tests. The predictor must handle real vehicles, microphones, speakers, and ambient noise without demanding needless repetitions. Synthetic corruption also does not cover every failure that can appear in use, such as overlapping speakers, unusual accents, clipped speech, or microphone problems specific to a device.
The trade-off is unavoidable: flag too much and people repeat themselves unnecessarily; flag too little and the system still misinterprets them with confidence. The paper’s larger point is that a speech interface should be able to stop before answering when its own audio model is unlikely to have heard the words correctly. If the result holds on real recordings and can be calibrated for each deployed model, asking for a repeat could become a built-in part of reliable voice systems rather than a check added after the wrong answer has already been made.