# Can AI Read the Room? USC Study Finds AI Is Better at Reading Than Listening > Source: > Published: 2026-08-10 17:22:43+00:00 Today, users can do more than exchange text with artificial intelligence (AI). From holding conversations and asking questions aloud to sending voice messages, AI can now process audios and respond in real time. But can AI really listen as well as it can read? A new study led by USC researchers suggests the answer is no. The team found that even today’s most advanced audio large language models (LLMs) struggle to interpret information beyond the spoken words, often taking language too literally and missing the nonverbal cues that have an equally important role in human communication. In everyday conversations, meaning extends far beyond words. Tone of voice, emotion, emphasis, pitch and other paralinguistic information provide important social and emotional context that helps people interpret what someone truly means. USC professor [Mohammad Soleymani](https://viterbi.usc.edu/directory/faculty/Soleymani/Mohammad) led the [research project](https://voxparadox.github.io/) to uncover why audio LLMs struggle to interpret these listening cues and developed new techniques that significantly improve their ability to understand the “how” of speech—not just the spoken words. The project, which began last August, resulted in the paper, “[Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox](https://openreview.net/challenge?redirect=%2Fforum%3Fid%3Dv7rYbRR9Zw),” which was accepted to the [International Conference on Machine Learning (ICML) 2026](https://icml.cc/). Soleymani is a research associate professor at [USC Viterbi School of Engineering](https://viterbi.usc.edu/) and the [USC Mark and Mary Stevens School of Computing and AI](https://stevens-computing-ai.usc.edu/)‘s [Thomas Lord Department of Computer Science](https://www.cs.usc.edu/), with joint appointment at the [USC Institute for Creative Technologies](https://ict.usc.edu/). He also leads the USC [Intelligent Human Perception (IHP) Lab](https://ihp-lab.org/). The research team also included Soleymani’s PhD students Ashutosh Chaubey, and Jiacheng Pang, a former master’s student from his lab. ## Why AI Models Struggle to Listen Audio LLMs are multimodal AI models that can process not only text but also audio inputs, such as speech and environmental sounds. Embedded in conversational AI tools like ChatGPT and Gemini, Audio LLMs allow users to speak directly to AI, receive spoken responses in real time and analyze uploaded audio recordings. Audio LLMs work by converting speech into numerical representations, known as vectors, that are translated into a format the model’s internal “language brain” can understand. While these models are designed to be multimodal and process both spoken content and acoustic information, they suffer from a strong language bias, “treating text as a first-class citizen and audio cues as secondary,” Soleymani explained. Because the models are so focused on language, they often ignore the acoustic reality of an audio clip. As a result, audio LLMs excel at transcribing what is said, accurately converting spoken words into text, but often struggle to understand how something is said—such as the speaker’s tone, emotion or pitch, making them functionally “tone-deaf.” For example, if a speaker sounds clearly happy but says the words, “I am sad,” the model will incorrectly conclude that the speaker is sad. This can lead to misinterpretations of a speaker’s intent, emotional state and social context. In sensitive applications such as health assessments or human-AI interactions, failing to “hear” a user’s true emotion or condition could result in incorrect or even harmful responses. ## Testing AI With Contradictory Audio Tasks After identifying this blind spot in Audio LLMs, Soleymani’s team designed a series of “stress tests” to evaluate how well the models understood paralinguistic information. The researchers introduced a benchmark called VoxParadox, which evaluates models across 10 different paralinguistic tasks. These tasks include biometric and identity-related challenges, such as estimating a speaker’s age or gender and counting how many people are speaking, as well as prosodic and acoustic tasks, including identifying emotion, intonation, pitch and volume. For each task, the researchers presented the models with audio clips where spoken words intentionally contradicted the accompanying audio content and asked them to determine whether the uttered words were true or false. If the model answered incorrectly, it suggested the model was prioritizing the text over the audio. If it answered correctly, it demonstrated that the model had incorporated the nonverbal audio information into its reasoning. The team evaluated NVIDIA’s Audio Flamingo 3 and Alibaba’s Qwen2-Audio models and found that both struggled to accurately interpret paralinguistic information. ## Looking Inside AI’s Brain Soleymani’s team then set out to understand why Audio LLMs fail at interpreting paralinguistic information by probing the models’ internal layers—almost like performing “brain surgery” on AI. By examining the models’ internal computations in real time, the researchers found that acoustic details, such as pitch and tone, gradually become “stripped away” as audio data moves through the model’s layers toward its language-processing components. By the time the information reaches the model’s final decision-making layer, much of the acoustic information has disappeared. The researchers refer to this phenomenon as “representation degradation.” The team also identified what they call a “utilization gap.” Although the correct acoustic information often still exists within the model’s internal representations, the model’s decision-making process is so heavily biased toward language that it simply ignores the available audio evidence. ## Giving AI Hearing Aids & Training AI to Read the Room The team proposed a two-part solution designed to help Audio LLMs both retain and value what they hear. First, the researchers developed a module called the Prompt-Conditioned Layer Mixer (PCLM), which helps the model determine exactly which part of the audio it should “listen” to based on the question it is being asked. As audio passes through an encoder, acoustic details such as pitch and tone are often suppressed in the deeper layers as the model becomes increasingly focused on words. Instead of relying only on the encoder’s final layer, PCLM draws information from multiple layers simultaneously before passing it to the language model. This allows the model to access both the early layers of processing—which are better at capturing basic acoustic information such as pitch, volume and speaker characteristics—and the deeper layers, which specialize in understanding vocabulary and language. As a result, the model can revisit those early “hearing” layers when answering questions about how someone sounds, rather than relying only on the language-focused layers that interpret what was said. PCLM also examines the user’s text prompt to determine the optimal combination of audio information needed for a particular task. Soleymani describes this as a “soft selection” process, similar to following a recipe. For example, if a user asks, “Is this speaker angry?” PCLM recognizes that the task depends more heavily on emotional cues than on the spoken words. It may therefore draw more information from the early acoustic-processing layers and less from the language-focused layers to make a more accurate judgment. “Just like a hearing aid, PCLM boosts the volume of the audio cues that are most relevant to your question,” Soleymani said. The team also applied a second technique called Direct Preference Optimization (DPO), a post-training method that aligns the model’s decision-making process with the available acoustic evidence. During DPO training, the model is shown pairs of responses and learns which answer should be preferred because it better matches the audio, encouraging it to rely less on textual shortcuts and more on what it actually hears. Together, PCLM and DPO produced “very significant improvements,” according to the researchers. After incorporating both methods, the models’ ability to interpret paralinguistic information improved dramatically, with accuracy increasing from 17% to 65%. The study is the first to both identify this blind spot in Audio LLMs and propose solutions to address it. “By identifying this new problem, we open the door for other researchers to recognize it, study it and develop even better solutions,” Soleymani said. “It creates awareness of a blind spot that the field hadn’t fully understood before.” Published on August 10th, 2026 Last updated on August 10th, 2026