Rethinking Gender Annotation for Bias Evaluation in Machine Translation: Can LLMs Improve Reliability?

Abstract

The assessment of gender bias in Machine Translation critically depends on reliable methods for identifying grammatical gender in system outputs. In this paper, we compare the automated WinoMT annotation pipeline, based on word alignment and morphological tagging, with an instruction-tuned LLM (Qwen3-8B) used to annotate grammatical gender in English–Italian translations. Both approaches achieve similar levels of agreement with a human-annotated gold standard, but exhibit distinct systematic weaknesses. The WinoMT pipeline is sensitive to alignment shifts and morphological tagging limitations, often resulting in indeterminate gender labels. The LLM, in contrast, tends to favour binary gender labels even when noun phrases are morphologically gender-invariant. A qualitative analysis of model outputs and reasoning traces further suggests a lack of stable generalization of grammatical gender rules, even when illustrative exemplars are provided for in-context learning. This highlights important methodological limitations in using LLMs as objective annotators for gender evaluation tasks.

- Anthology ID:
- 2026.gitt-1.3
- Volume:
- [Proceedings of the 4th Workshop on Gender-Inclusive Translation Technologies (GITT 2026)](https://aclanthology.org/volumes/2026.gitt-1/)
- Month:

• June
• Year:
• 2026
• Address:
• Tilburg, the Netherlands
• Editors:
• Manuel Lardelli ,Beatrice Savoldi ,Janiça Hackenbuchner ,Luisa Bentivogli ,Eleni Gkovedarou ,Joke Daems

- Venues:
- [GITT](https://aclanthology.org/venues/gitt/) |[WS](https://aclanthology.org/venues/ws/)
- SIG:
- Publisher:

• European Association for Machine Translation

- Note:
- Pages:

• 16–30

- Language:
- URL:
- [https://aclanthology.org/2026.gitt-1.3/](https://aclanthology.org/2026.gitt-1.3/)
- DOI:
- Cite (ACL):

• Chiara Manna, Argentina Anna Rescigno, and Eva Vanmassenhove. 2026. Rethinking Gender Annotation for Bias Evaluation in Machine Translation: Can LLMs Improve Reliability? . InProceedings of the 4th Workshop on Gender-Inclusive Translation Technologies (GITT 2026) , pages 16–30, Tilburg, the Netherlands. European Association for Machine Translation.

- Cite (Informal):
- [Rethinking Gender Annotation for Bias Evaluation in Machine Translation: Can LLMs Improve Reliability?](https://aclanthology.org/2026.gitt-1.3/) (Manna et al., GITT 2026)
- PDF:
- [https://aclanthology.org/2026.gitt-1.3.pdf](https://aclanthology.org/2026.gitt-1.3.pdf)