Hello Hugging Face folks,
I’ve developed an open-source Python project demonstrating a practical workflow for LLM response evaluation and annotation quality assurance.
The benchmark covers:
Weighted response scoring across four dimensions.
Structured annotation validation using Pydantic.
A shared error taxonomy.
Evaluator agreement and disagreement analysis.
Human adjudication for disputed cases.
The sample pipeline includes 15 shared evaluation cases, 86.7% error-label agreement, 93.3% severity agreement and 3 cases flagged for human review.
I’m interested in feedback from practitioners working on LLM evaluation, RLHF, data quality and annotation workflows.
Some questions I’d particularly appreciate feedback on:
Which additional evaluation dimensions would you recommend?
What is the best approach to extend this into RAG factuality and citation correctness evaluation?
Would integration with Label Studio or Argilla be useful?
What additional metrics should be included in evaluator agreement reporting?
Dataset: goodgentleman918/ai-data-quality-llm-evaluation-benchmark · Datasets at Hugging Face Contributions and constructive suggestions are welcome.