cd /news/ai-safety/no-harm-jury-a-jury-based-multi-dime… · home › topics › ai-safety › article
[ARTICLE · art-173016] src=arxiv.org ↗ pub= topic=ai-safety verified=true sentiment=· neutral

No-Harm-Jury A Jury-Based Multi-Dimensional Harm Evaluation Framework for Medical AI Responses

A new arXiv paper (2611.24972v1) presents No-Harm-Jury, a jury-based automated evaluation framework that scores AI-generated medical responses across seven harm dimensions using a panel of five diverse language models as independent jurors. The framework scored responses generated by Ministral-14B on 3,000 questions from PubMedQA, MedQA, and MedMCQA, excluding 151 empty responses (98 MedQA, 53 MedMCQA) to leave 2,849 valid samples, with median aggregation used to suppress individual juror biases. The authors state no existing system combines all five properties of No-Harm-Jury: automated scalable evaluation, five diverse LLM jurors, robust median aggregation with a reliability-weighted extension, seven conceptually distinct harm dimensions, and application to medical QA benchmarks.

recommendation

When evaluating medical AI responses, use a panel of five diverse LLM jurors and aggregate with the median rather than a single judge to suppress individual juror bias across the seven harm dimensions.

more in playbook: Evals and testing for LLM apps
by read1 min views3 publishedNov 20, 2026

arXiv:2611.24972v1 Announce Type: new Abstract: We present No-Harm-Jury, a jury-based automated evaluation framework that scores AI-generated medical responses across seven conceptually distinct harm dimensions using a panel of five diverse language models as independent jurors. Responses generated by Ministral-14B on 3,000 questions drawn from PubMedQA, MedQA, and MedMCQA are scored by five jurors across dimensions of informational, social, psychological, autonomy, economic, privacy, and epistemic harm; the 151 empty responses (98 MedQA, 53 MedM- CQA) are excluded, leaving 2,849 valid samples. The median aggregation robustly suppresses individual juror biases while maintaining coverage through complementary specialisation. No existing system combines all five prop- erties of No-Harm-Jury: automated scalable evaluation, five diverse LLM jurors, robust me- dian aggregation (with a reliability-weighted extension), seven conceptually distinct harm dimensions, and application to medical QA benchmarks.

── more in #ai-safety 4 stories · sorted by recency
── more on @no-harm-jury 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/no-harm-jury-a-jury-…] indexed:0 read:1min 2026-11-20 · —