{"slug": "inclusionmed-index-a-healthcare-ai-benchmark-that-ranks-models-by-role", "title": "InclusionMed Index: A Healthcare AI Benchmark That Ranks Models by Role", "summary": "InclusionMed, a November 2026 open healthcare evaluation initiative, released a technical report on November 16, 2026 scoring 22 frontier language models across five audited benchmarks, with overall Index values ranging from 17.5 to 49.1 out of 100. The project audited 388 candidate benchmarks, advanced four public benchmarks plus the private InclusionMed-DxFiles diagnostic reasoning suite, and organizes results across a 3-role by 3-capability grid covering three of nine Atlas cells, so rankings shift when filtered by clinician, patient, or administrator role. The report, project site, and GitHub repository ship together, giving builders a way to compare models for triage chat, prior authorization, and clinical documentation before wiring them into Epic, Cerner, or payer portals.", "body_md": "**InclusionMed is a November 2026 open healthcare evaluation initiative that scores frontier language models on five audited benchmarks — and reports overall Index values from 17.5 to 49.1 out of 100, with rankings that change when you filter by clinician, patient, or administrator roles.** The [arXiv technical report](https://arxiv.org/abs/2611.21218?ref=explainx), [project site](https://inclusionmed.ai/?ref=explainx), and [GitHub repository](https://github.com/panuvi/inclusionmed?ref=explainx) ship together.\n\nFor builders picking models for **triage chat, prior auth, or clinical documentation**, the headline is not a single leaderboard number. It is that **the best aggregate model may still be the wrong choice for the hat your agent wears.**\n\n## TL;DR\n\n| Question | Answer | \n|---|---|\n| What released? | Technical report Nov 16, 2026; HF Paper of the Day Nov 17 | \n| Scope | 22 models, 5 benchmarks, 388 candidates audited | \n| Score band | 17.5–49.1 overall Index | \n| Structure | 3 healthcare roles × 3 AI capabilities grid (9 cells) | \n| Open artifacts | Atlas, Index, contribution guides on GitHub | \n\n## Why inclusive healthcare evaluation needed a new atlas\n\nMedical AI marketing still leans on **single-number leaderboards** from narrow exam-style tests. Real care settings split work across **patients**, **clinicians**, and **administrators**, and expect different **knowledge, reasoning, and agency** skills in each lane.\n\nInclusionMed's response is structural:\n\n1. **InclusionMed Atlas** — maps which benchmarks occupy which role-capability cell.\n2. **InclusionMed Index** — equally weighted mean across included benchmarks, plus slices by role and capability.\n3. **Community contributions** — audited intake so future releases widen coverage instead of swapping one vanity metric for another.\n\nThe release landed the same week as [OpenAI's healthcare computer-use agents post](https://explainx.ai/blog/openai-computer-use-healthcare-gpt-6-astra-cvs-november-2026) and [Google's Gemini for Healthcare preview](https://explainx.ai/blog/google-gemini-healthcare-preview-skills-connectors-november-2026). Those announcements describe **product pilots**; InclusionMed describes **how to compare models** before you wire them into Epic, Cerner, or payer portals.\n\n## How benchmarks enter the Atlas\n\nClinical documentation motif linking healthcare evaluation to structured medical AI benchmarks\n\nThe GitHub README lists **six audit principles** applied to every candidate, public or private:\n\n| Principle | What reviewers check | \n|---|---|\n| Healthcare relevance | Tasks map to real care activities | \n| Trustworthiness | Ground truth and scoring have professional basis | \n| Availability | Tasks, code, and licenses permit evaluation | \n| Complementary coverage | Fills gaps in roles, languages, or settings | \n| Timeliness | Content still matches current guidelines | \n| Discriminative capacity | Scores separate models under fair conditions | \n\nFrom **388 candidates**, the team advanced **four public benchmarks** through structured audits and **independent reproduction**, then added **InclusionMed-DxFiles**, a **private** diagnostic reasoning suite, for five total tasks covering **three of nine** Atlas cells.\n\nHatched cells in project figures mark **intentional gaps** — places where inclusive healthcare still lacks a trusted test. That honesty is more useful than padding the Index with duplicate MedQA clones.\n\n## What the first Index numbers show\n\nThe paper's headline range — **17.5 to 49.1** on a 100-point equally weighted mean — signals that **frontier models are far from saturated** on this portfolio, even when vendors market near-perfect clinical demos.\n\nTwo subtler findings matter for product teams:\n\n### Rankings move when you change the hat\n\nThe Index publishes scores **by healthcare role** and **by AI capability**. A model leading the overall table can fall mid-pack on **administrative agency** or **professional reasoning** depending on the slice.\n\nThat behavior should inform routing in multi-agent clinic stacks: a cheap model tuned for **patient education** may beat an expensive general model on bedside tone while failing structured utilization review.\n\n### Patient communication beat clinical decision-making among leaders\n\nWithin-benchmark **task-level tags** show an uncomfortable pattern: for the **ten highest Index models**, **patient communication scores exceeded clinical decision-making scores** in every case.\n\nIf you ship a chat surface to discharged patients, benchmark that lane explicitly — do not assume a high USMLE-style number protects prescribing suggestions hidden in the same weights.\n\n### Cost diverges at similar Index scores\n\nThe project's **cost–performance frontier** chart (on inclusionmed.ai) plots estimated **USD per task** from recorded token usage and published API prices. Models with **similar Index scores** can sit orders of magnitude apart on cost.\n\nFor health systems watching [OpenAI Luna pricing in computer-use benchmarks](https://explainx.ai/blog/openai-computer-use-healthcare-gpt-6-astra-cvs-november-2026) or running [Apple's health AI usage studies](https://explainx.ai/blog/apple-health-ai-usage-study-openevidence-2026), InclusionMed reinforces: **pick per workflow**, not per press release.\n\n## InclusionMed-DxFiles and the private benchmark choice\n\nThe Atlas marks **InclusionMed-DxFiles** with a dagger as a **private diagnostic reasoning benchmark**. Public reproducibility fans may chafe, but the authors argue some clinical items cannot ship openly without contaminating future evaluations.\n\nBuilders should note:\n\n- Public leaderboard claims that omit private cells are **incomplete** relative to the full Index.\n- Contribution pathways exist to add **new public suites** that fill hatched cells without waiting for the next corporate benchmark drop.\n\n## How to use InclusionMed this week\n\n1. **Map your agent's job** to a role-capability cell before you trust any single score.\n2. **Open the interactive Atlas** on[inclusionmed.ai](https://inclusionmed.ai/?ref=explainx) and record which benchmarks actually ran for your shortlisted models.\n3. **Clone the GitHub checklist** ([evaluation-record requirements](https://github.com/panuvi/inclusionmed/blob/main/inclusionmed-evaluation-checklist.md?ref=explainx) ) into your internal eval CI so regressions are comparable release to release.\n4. **Pair Index scores with live EHR tests** — benchmarks do not replace[Epic-connected ChatGPT pilots](https://explainx.ai/blog/chatgpt-healthcare-epic-ehr-integration-2026) or HIPAA BAA constraints.\n\nIf you fine-tune open weights for chart summarization, compare InclusionMed slices with [Sarvam's India-focused inference pricing](https://explainx.ai/blog/sarvam-token-factory-india-open-weight-inference-rupee-pricing-2026) or other regional hosts — the Index is model-centric, not deployment-centric.\n\n## Contributing and staying current\n\nThe collaboration invites four contribution types on GitHub:\n\n- **Benchmarks** with auditable materials\n- **Benchmark ideas** when you see a care gap but no dataset yet\n- **Corrections** with evidence for Index rows\n- **Advisory partnerships** scoped before announcement\n\nContact: **[info@inclusionmed.ai](mailto:info@inclusionmed.ai)** per the repository README.\n\nFor governance teams, InclusionMed complements policy news like [OpenAI CSAM classifier warnings](https://explainx.ai/blog/openai-csam-image-classifier-chatgpt-warnings-november-2026) — one addresses **clinical fitness**, the other **platform abuse**. Both belong in enterprise risk reviews.\n\n## Limits to keep in mind\n\n- **Five benchmarks cannot represent global healthcare** ; hatched cells are explicit admission of that fact.\n- **Private DxFiles** means external teams cannot fully replicate the published Index without collaboration.\n- **Snapshot models** — scores reflect November 2026 frontier weights; weekly vendor updates can reorder slices.\n\nTreat InclusionMed as a **directional compass**, then run **your own tasks** on de-identified charts and payer PDFs.\n\n## Walking through the six audit principles in practice\n\nThe GitHub README's audit rubric is the backbone of InclusionMed's credibility claim. Here is how each principle changes what lands in the Atlas:\n\n**Healthcare relevance** filters vanity NLP tasks dressed in scrubs. A benchmark must tie items to identifiable care work — triage questions, utilization management, patient education — not generic multiple-choice trivia with medical vocabulary sprinkled in.\n\n**Trustworthiness** is where many public sets fail quietly. Reviewers examine annotation manuals, clinician involvement, and whether reference answers trace to guidelines or committee consensus. The initiative excludes benchmarks generated entirely by models without human professional review — a bar more suites should adopt.\n\n**Availability and permitted use** stops the common trap of leaderboard papers that cite private items readers cannot run. InclusionMed still keeps **DxFiles** private, but marks it visibly; transparency about that choice is part of the audit.\n\n**Complementary coverage** forces each new candidate to justify its cell in the 3×3 grid. Otherwise the Index would re-weight the same MedQA derivatives five times and call it inclusivity.\n\n**Timeliness and data independence** matter because clinical guidelines shift — what was true for COVID triage in 2021 is not the right lens for 2026 agent copilots. Reviewers look for contamination guards when models may have trained on test items.\n\n**Discriminative capacity** uses published score spreads to reject benchmarks where every frontier model ties. If nothing separates, the benchmark does not help procurement.\n\n## How InclusionMed differs from single-suite healthcare leaderboards\n\n| Approach | InclusionMed | Typical vendor healthcare blog | \n|---|---|---|\n| Unit of score | Role-capability slices + overall Index | One pilot metric (e.g., inbox safety %) | \n| Model count | 22 in first release | Often one flagship model | \n| Cost | Published frontier on cost–performance chart | Sometimes omitted | \n| Gaps | Hatched Atlas cells | Often unmentioned | \n\nOpenAI's **99% clinical safety** statistic on EHR inbox work is invaluable for **deployment narrative** — read it alongside InclusionMed when you ask whether the same model leads on **administrative agency** or **patient-only chat**.\n\n## Governance hooks for hospital innovation teams\n\n- Add InclusionMed slice requirements to **vendor RFPs** when a startup claims \"SOTA clinical AI.\"\n- Mirror the [evaluation-record checklist](https://github.com/panuvi/inclusionmed/blob/main/inclusionmed-evaluation-checklist.md?ref=explainx) in your MLOps pipeline so fine-tunes do not ship without role-specific regression.\n- Pair Atlas gaps with internal task libraries — if the **administrator × agency** cell is empty, your billing agent needs a bespoke eval, not a generic chat guardrail.\n\nRegulatory readers may also crosswalk with [California's November litigation against model hosts](https://explainx.ai/blog/california-ag-bonta-sues-openai-hugging-face-lawsuit-november-2026): benchmarks do not replace safety testing, but they document **fit-for-purpose** claims procurement lawyers can audit.\n\nSchedule quarterly reruns when vendors ship new healthcare-tuned weights; the Index is a snapshot, not a perpetual certificate.\n\nWhen you cite InclusionMed in board slides, show the **role slice** that matches the workflow you are buying, not only the overall Index bar chart.", "url": "https://wpnews.pro/news/inclusionmed-index-a-healthcare-ai-benchmark-that-ranks-models-by-role", "canonical_source": "https://explainx.ai/blog/inclusionmed-healthcare-ai-benchmark-index-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog", "published_at": "2026-11-17 00:00:00+00:00", "updated_at": "2026-11-17 20:10:30.846222+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-safety", "ai-products"], "entities": ["InclusionMed", "InclusionMed Index", "InclusionMed Atlas", "InclusionMed-DxFiles", "GitHub", "Epic", "Cerner", "arXiv"], "also_reported_by": [], "takeaway": "Across 22 models and five audited benchmarks, InclusionMed Index scores span 17.5 to 49.1 out of 100, and a model leading the overall table can fall mid-pack on administrative agency or professional reasoning, so teams should route by role slice rather than aggregate rank.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/inclusionmed-index-a-healthcare-ai-benchmark-that-ranks-models-by-role", "markdown": "https://wpnews.pro/news/inclusionmed-index-a-healthcare-ai-benchmark-that-ranks-models-by-role.md", "text": "https://wpnews.pro/news/inclusionmed-index-a-healthcare-ai-benchmark-that-ranks-models-by-role.txt", "jsonld": "https://wpnews.pro/news/inclusionmed-index-a-healthcare-ai-benchmark-that-ranks-models-by-role.jsonld"}}