Abstract
Human evaluation is the gold standard for Natural Language Generation (NLG), but the tooling is fragmented: most platforms serve a single task or a single protocol, and none treat the growing practice of assisting annotators with an LLM as a measurable object of study. These gaps fall hardest on indigenous and under-resourced languages, where paid crowds are scarce, references are unavailable, and data governance matters. We present Aura, an open, self-hostable platform that unifies five evaluation protocols (Likert/DA, pairwise, ranking, error-span, post-editing) across eight task families under one segment schema and one result model, records writing-system and speaker provenance as first-class fields, and runs entirely offline. Its distinguishing feature is a grounded, advisory-only, logged LLM assistant paired with a built-in randomised experiment that estimates the effect of AI assistance on inter-annotator agreement and time-on-task. Aura’s agreement statistics match reference implementations to machine precision. We conducted a human evaluation experiment on three English→Indian-language ranking campaigns (N=9 annotators) which returned a mean usability rating of 4.06/5. Aura is released as open-source software.
- Anthology ID:
- 2026.aacl-demo.2
- Volume:
- Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations
- Month:
- November
- Year:
- 2026
- Address:
- Hengqin, China
- Editors:
- [Ningyu Zhang](https://aclanthology.org/people/ningyu-zhang/) ,[Derry Wijaya](https://aclanthology.org/people/derry-tanti-wijaya/)
- Venues:
- [AACL](https://aclanthology.org/venues/aacl/) |[IJCNLP](https://aclanthology.org/venues/ijcnlp/)
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 9–16
- Language:
- URL:
- [https://aclanthology.org/2026.aacl-demo.2/](https://aclanthology.org/2026.aacl-demo.2/)
- DOI:
- Cite (ACL):
- Rudali Huidrom, Khangembam Alex D Nelson, Nicholas Khumukcham, Hoomexsun Pangsatabam, Jesu Nazarene Roy L, Kanal Trivedi, Vikas Kumar, Pinaki Das, Kshetrimayum Boynao Singh, Tarun Bisht, Justice Konjengbam, Konjengbam Anand, and Vikram Goyal. 2026. Aura 💫: A Self-Hostable Platform for Human Evaluation of NLG for Under-resourced Languages and Beyond . InProceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations , pages 9–16, Hengqin, China. Association for Computational Linguistics.
- Cite (Informal):
- [Aura 💫: A Self-Hostable Platform for Human Evaluation of NLG for Under-resourced Languages and Beyond](https://aclanthology.org/2026.aacl-demo.2/) (Huidrom et al., AACL-IJCNLP 2026)
- PDF:
- [https://aclanthology.org/2026.aacl-demo.2.pdf](https://aclanthology.org/2026.aacl-demo.2.pdf)