Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions

Researchers introduced Indi-RomCoM, a benchmark for evaluating large language models on Romanized code-mixed Indic-English instructions. Testing across seven tasks and four languages showed that LLMs consistently underperform as code-mixing density increases, with reasoning tasks degrading less than detection tasks.

arXiv:2606.30790v1 Announce Type: new Abstract: Romanized Code Mixing RCM , where bilingual speakers fluidly blend local languages with English in Roman script, has emerged as the dominant form of communication across multilingual communities. While Large Language Models LLMs perform strongly on monolingual and native-script benchmarks, their ability to follow instructions and reason over RCM-based content remains largely unexplored. To this end, we introduce the Indi-RomCoM benchmark for facilitating systematic evaluation on Indic Romanized Code-Mixed instructions. Our benchmark spans seven instruction-following tasks, four widely spoken Indic languages, and three controlled code-mixing intensity levels. We extensively evaluate a suite of LLMs covering proprietary, open-weight, and Indic-focused models under zero- and few-shot settings. LLMs consistently underperform on RCM instructions, with performance degrading as code-mixing density increases. Furthermore, reasoning tasks suffer less degradation than detection tasks e.g., Toxicity because the generated explanations offer necessary context. We believe Indi-RomCoM helps the community in developing inclusive multilingual systems.