{"slug": "aura-a-self-hostable-platform-for-human-evaluation-of-nlg-for-under-resourced", "title": "Aura 💫: A Self-Hostable Platform for Human Evaluation of NLG for Under-resourced Languages and Beyond", "summary": "Researchers led by Rudali Huidrom released Aura, an open-source, self-hostable platform that unifies five human-evaluation protocols (Likert/DA, pairwise, ranking, error-span, post-editing) across eight task families for Natural Language Generation, with a built-in randomized experiment measuring how an advisory-only LLM assistant affects inter-annotator agreement and time-on-task. In a human evaluation of three English→Indian-language ranking campaigns with N=9 annotators, Aura returned a mean usability rating of 4.06/5, and its agreement statistics matched reference implementations to machine precision. The work appears in the AACL-IJCNLP 2026 System Demonstrations proceedings, pages 9–16, and targets indigenous and under-resourced languages where paid crowds and references are scarce and data governance matters.", "body_md": "##### Abstract\n\nHuman evaluation is the gold standard for Natural Language Generation (NLG), but the tooling is fragmented: most platforms serve a single task or a single protocol, and none treat the growing practice of assisting annotators with an LLM as a measurable object of study. These gaps fall hardest on indigenous and under-resourced languages, where paid crowds are scarce, references are unavailable, and data governance matters. We present Aura, an open, self-hostable platform that unifies five evaluation protocols (Likert/DA, pairwise, ranking, error-span, post-editing) across eight task families under one segment schema and one result model, records writing-system and speaker provenance as first-class fields, and runs entirely offline. Its distinguishing feature is a grounded, advisory-only, logged LLM assistant paired with a built-in randomised experiment that estimates the effect of AI assistance on inter-annotator agreement and time-on-task. Aura’s agreement statistics match reference implementations to machine precision. We conducted a human evaluation experiment on three English→Indian-language ranking campaigns (N=9 annotators) which returned a mean usability rating of 4.06/5. Aura is released as open-source software.\n- Anthology ID:\n- 2026.aacl-demo.2\n- Volume:\n- [Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations](https://aclanthology.org/volumes/2026.aacl-demo/)\n- Month:\n- November\n- Year:\n- 2026\n- Address:\n- Hengqin, China\n- Editors:\n- [Ningyu Zhang](https://aclanthology.org/people/ningyu-zhang/) ,[Derry Wijaya](https://aclanthology.org/people/derry-tanti-wijaya/)\n- Venues:\n- [AACL](https://aclanthology.org/venues/aacl/) |[IJCNLP](https://aclanthology.org/venues/ijcnlp/)\n- SIG:\n- Publisher:\n- Association for Computational Linguistics\n- Note:\n- Pages:\n- 9–16\n- Language:\n- URL:\n- [https://aclanthology.org/2026.aacl-demo.2/](https://aclanthology.org/2026.aacl-demo.2/)\n- DOI:\n- Cite (ACL):\n- Rudali Huidrom, Khangembam Alex D Nelson, Nicholas Khumukcham, Hoomexsun Pangsatabam, Jesu Nazarene Roy L, Kanal Trivedi, Vikas Kumar, Pinaki Das, Kshetrimayum Boynao Singh, Tarun Bisht, Justice Konjengbam, Konjengbam Anand, and Vikram Goyal. 2026. [Aura 💫: A Self-Hostable Platform for Human Evaluation of NLG for Under-resourced Languages and Beyond](https://aclanthology.org/2026.aacl-demo.2/) . In*Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations* , pages 9–16, Hengqin, China. Association for Computational Linguistics.\n- Cite (Informal):\n- [Aura 💫: A Self-Hostable Platform for Human Evaluation of NLG for Under-resourced Languages and Beyond](https://aclanthology.org/2026.aacl-demo.2/) (Huidrom et al., AACL-IJCNLP 2026)\n- PDF:\n- [https://aclanthology.org/2026.aacl-demo.2.pdf](https://aclanthology.org/2026.aacl-demo.2.pdf)", "url": "https://wpnews.pro/news/aura-a-self-hostable-platform-for-human-evaluation-of-nlg-for-under-resourced", "canonical_source": "https://aclanthology.org/2026.aacl-demo.2/", "published_at": "2026-11-02 00:00:00+00:00", "updated_at": "2026-11-05 02:19:33.514413+00:00", "lang": "en", "topics": ["natural-language-processing", "ai-research", "ai-tools", "large-language-models"], "entities": ["Aura", "Rudali Huidrom", "Association for Computational Linguistics", "AACL-IJCNLP 2026", "Khangembam Alex D Nelson", "Vikram Goyal"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/aura-a-self-hostable-platform-for-human-evaluation-of-nlg-for-under-resourced", "markdown": "https://wpnews.pro/news/aura-a-self-hostable-platform-for-human-evaluation-of-nlg-for-under-resourced.md", "text": "https://wpnews.pro/news/aura-a-self-hostable-platform-for-human-evaluation-of-nlg-for-under-resourced.txt", "jsonld": "https://wpnews.pro/news/aura-a-self-hostable-platform-for-human-evaluation-of-nlg-for-under-resourced.jsonld"}}