{"slug": "faithful-by-design-evaluating-and-improving-llm-generated-clinical-trial-for", "title": "Faithful by Design: Evaluating and Improving LLM-Generated Clinical Trial Summaries for Multi-Stakeholder Audiences", "summary": "Researchers introduced a benchmark to evaluate faithfulness of LLM-generated clinical trial summaries for healthcare providers, patients, and payers, finding that GPT-4o, Claude Sonnet 4.6, and Gemini 2.5 Flash frequently produced unsupported claims. A knowledge-graph-augmented retrieval system improved faithfulness scores across all models, with statistically significant gains in entailment and overall faithfulness.", "body_md": "arXiv:2607.09932v1 Announce Type: new\nAbstract: Large language models are increasingly used to summarize clinical trial results for healthcare providers, patients, and payers, but their tendency to hallucinate poses significant risks in this high-stakes context. This study introduces a benchmark evaluation framework for measuring the faithfulness of LLM-generated clinical trial summaries across three stakeholder audiences. The framework consists of 200 stratified trials drawn from the Aggregate Analysis of ClinicalTrials.gov database, evaluated using audience-specific prompt templates and a six-dimension faithfulness annotation schema. Baseline measurements were established for GPT-4o, Claude Sonnet 4.6, and Gemini 2.5 Flash across 1,800 generated summaries scored using a cross-encoder natural language inference (NLI) model. Unsupported Claims was identified as the dominant failure mode across all three models, with a mean annotation score of 1.55 out of three. A knowledge-graph-augmented retrieval system was developed and evaluated against the baseline, producing statistically significant improvements in NLI-based faithfulness scores (entailment +0.0125, faithfulness +0.0130, p < 0.0001). Improvement pathways were model-dependent, with GPT-4o improving primarily through contradiction reduction while Claude Sonnet 4.6 and Gemini 2.5 Flash improved through increased entailment.", "url": "https://wpnews.pro/news/faithful-by-design-evaluating-and-improving-llm-generated-clinical-trial-for", "canonical_source": "https://arxiv.org/abs/2607.09932", "published_at": "2026-07-14 04:00:00+00:00", "updated_at": "2026-07-14 04:33:36.835350+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-research", "natural-language-processing"], "entities": ["GPT-4o", "Claude Sonnet 4.6", "Gemini 2.5 Flash", "Aggregate Analysis of ClinicalTrials.gov"], "alternates": {"html": "https://wpnews.pro/news/faithful-by-design-evaluating-and-improving-llm-generated-clinical-trial-for", "markdown": "https://wpnews.pro/news/faithful-by-design-evaluating-and-improving-llm-generated-clinical-trial-for.md", "text": "https://wpnews.pro/news/faithful-by-design-evaluating-and-improving-llm-generated-clinical-trial-for.txt", "jsonld": "https://wpnews.pro/news/faithful-by-design-evaluating-and-improving-llm-generated-clinical-trial-for.jsonld"}}