Behind the Smokescreen: Uncovering Cross-LLM Bias and Reasoning Contradictions of LLM-as-a-judge for Generated Feedback

Céline Leuzinger published a study in the Proceedings of the 15th Workshop on Natural Language Processing for Computer Assisted Language Learning (NLP4CALL 2026, pages 269–287, Gothenburg, Sweden) examining cross-LLM bias and reasoning contradictions when large language models act as judges of generated feedback. The paper, titled "Behind the Smokescreen: Uncovering Cross-LLM Bias and Reasoning Contradictions of LLM-as-a-judge for Generated Feedback," was released through the University of Tartu Library and is available via the ACL Anthology (2026.nlp4call-1.20). The work matters because LLM-as-a-judge setups are widely used to evaluate generated feedback, and the study documents bias and contradictions across models that undermine that evaluation method.

- Anthology ID: - 2026.nlp4call-1.20 - Volume: - Proceedings of the 15th Workshop on Natural Language Processing for Computer Assisted Language Learning https://aclanthology.org/volumes/2026.nlp4call-1/ - Month: - November - Year: - 2026 - Address: - Gothenburg, Sweden - Editors: - Arianna Masciolini https://aclanthology.org/people/arianna-masciolini/ , David Alfter https://aclanthology.org/people/david-alfter/ , Herbert Lange https://aclanthology.org/people/herbert-lange/ , Ricardo Muñoz Sánchez https://aclanthology.org/people/ricardo-munoz-sanchez/ , Ildikó Pilán https://aclanthology.org/people/ildiko-pilan/unverified/ , Elena Volodina https://aclanthology.org/people/elena-volodina/ - Venues: - NLP4CALL https://aclanthology.org/venues/nlp4call/ | WS https://aclanthology.org/venues/ws/ - SIG: - SIGEDU https://aclanthology.org/sigs/sigedu/ - Publisher: - University of Tartu Library - Note: - Pages: - 269–287 - Language: - URL: - https://aclanthology.org/2026.nlp4call-1.20/ https://aclanthology.org/2026.nlp4call-1.20/ - DOI: - Cite ACL : - Céline Leuzinger. 2026. Behind the Smokescreen: Uncovering Cross-LLM Bias and Reasoning Contradictions of LLM-as-a-judge for Generated Feedback https://aclanthology.org/2026.nlp4call-1.20/ . In Proceedings of the 15th Workshop on Natural Language Processing for Computer Assisted Language Learning , pages 269–287, Gothenburg, Sweden. University of Tartu Library. - Cite Informal : - Behind the Smokescreen: Uncovering Cross-LLM Bias and Reasoning Contradictions of LLM-as-a-judge for Generated Feedback https://aclanthology.org/2026.nlp4call-1.20/ Leuzinger, NLP4CALL 2026 - PDF: - https://aclanthology.org/2026.nlp4call-1.20.pdf https://aclanthology.org/2026.nlp4call-1.20.pdf