Rethinking Gender Annotation for Bias Evaluation in Machine Translation: Can LLMs Improve Reliability?

A study by Chiara Manna, Argentina Anna Rescigno, and Eva Vanmassenhove compared the automated WinoMT annotation pipeline with the instruction-tuned LLM Qwen3-8B for annotating grammatical gender in English–Italian machine translation, finding both reach similar agreement with a human-annotated gold standard but fail in distinct ways. The WinoMT pipeline is sensitive to alignment shifts and morphological tagging limits, often producing indeterminate gender labels, while Qwen3-8B tends to favor binary gender labels even for morphologically gender-invariant noun phrases. The authors report that qualitative analysis of model outputs and reasoning traces suggests a lack of stable generalization of grammatical gender rules even with in-context exemplars, highlighting methodological limits of using LLMs as objective annotators for gender evaluation, in the paper published in the Proceedings of the 4th Workshop on Gender-Inclusive Translation Technologies (GITT 2026), pages 16–30.

Abstract The assessment of gender bias in Machine Translation critically depends on reliable methods for identifying grammatical gender in system outputs. In this paper, we compare the automated WinoMT annotation pipeline, based on word alignment and morphological tagging, with an instruction-tuned LLM Qwen3-8B used to annotate grammatical gender in English–Italian translations. Both approaches achieve similar levels of agreement with a human-annotated gold standard, but exhibit distinct systematic weaknesses. The WinoMT pipeline is sensitive to alignment shifts and morphological tagging limitations, often resulting in indeterminate gender labels. The LLM, in contrast, tends to favour binary gender labels even when noun phrases are morphologically gender-invariant. A qualitative analysis of model outputs and reasoning traces further suggests a lack of stable generalization of grammatical gender rules, even when illustrative exemplars are provided for in-context learning. This highlights important methodological limitations in using LLMs as objective annotators for gender evaluation tasks. - Anthology ID: - 2026.gitt-1.3 - Volume: - Proceedings of the 4th Workshop on Gender-Inclusive Translation Technologies GITT 2026 https://aclanthology.org/volumes/2026.gitt-1/ - Month: - June - Year: - 2026 - Address: - Tilburg, the Netherlands - Editors: - Manuel Lardelli https://aclanthology.org/people/manuel-lardelli/ , Beatrice Savoldi https://aclanthology.org/people/beatrice-savoldi/ , Janiça Hackenbuchner https://aclanthology.org/people/janica-hackenbuchner/ , Luisa Bentivogli https://aclanthology.org/people/luisa-bentivogli/ , Eleni Gkovedarou https://aclanthology.org/people/eleni-gkovedarou/ , Joke Daems https://aclanthology.org/people/joke-daems/ - Venues: - GITT https://aclanthology.org/venues/gitt/ | WS https://aclanthology.org/venues/ws/ - SIG: - Publisher: - European Association for Machine Translation - Note: - Pages: - 16–30 - Language: - URL: - https://aclanthology.org/2026.gitt-1.3/ https://aclanthology.org/2026.gitt-1.3/ - DOI: - Cite ACL : - Chiara Manna, Argentina Anna Rescigno, and Eva Vanmassenhove. 2026. Rethinking Gender Annotation for Bias Evaluation in Machine Translation: Can LLMs Improve Reliability? https://aclanthology.org/2026.gitt-1.3/ . In Proceedings of the 4th Workshop on Gender-Inclusive Translation Technologies GITT 2026 , pages 16–30, Tilburg, the Netherlands. European Association for Machine Translation. - Cite Informal : - Rethinking Gender Annotation for Bias Evaluation in Machine Translation: Can LLMs Improve Reliability? https://aclanthology.org/2026.gitt-1.3/ Manna et al., GITT 2026 - PDF: - https://aclanthology.org/2026.gitt-1.3.pdf https://aclanthology.org/2026.gitt-1.3.pdf