Are you speaking my languages? On spoken language adherence in multimodal LLMs

Researchers at arXiv propose a soft prompting approach to improve language adherence in multimodal LLMs for ASR, introducing a metric to quantify violations and evaluating zero-shot prompting, supervised fine-tuning, and Chain-of-Thought reasoning across multiple languages.

arXiv:2606.17281v1 Announce Type: new Abstract: While Large Language Model LLM based Automatic Speech Recognition ASR enables seamless multilingual use, models often misidentify the output language, compromising transcription fidelity and downstream application quality. To preserve flexibility and code-switching capabilities, we propose a soft prompting approach that hints at potential spoken languages without strictly constraining the output. We formally define this challenge as a lack of language adherence, introduce a novel metric to quantify violations, and evaluate three mitigation strategies: 1 zero-shot prompting for robust guidance under uncertainty, 2 supervised fine-tuning SFT to improve prompt adherence, and 3 Chain-of-Thought CoT reasoning to enforce adherence during decoding. We present a comparative analysis of these methods across multiple languages, evaluating effectiveness in reducing the language violation while maintaining overall ASR performance. Finally, we discuss trade-offs to guide strategy selection under various compute constraints.