Grounding Healthcare LLMs in a Causal Knowledge Graph: Framework, Metrics, and a Cardiovascular Pilot

A new arXiv preprint (arXiv:2608.15382v1) proposes a causal knowledge graph-based evaluation framework for healthcare large language models (LLMs), tested in a cardiovascular pilot. The framework's integrated grounding condition (C4) achieved the strongest causal edge F1 (0.838), adverse-effect F1 (0.833), evidence accuracy (0.738), and lowest unsupported claim rate (0.114), while the ungrounded condition (C1) had the highest raw intervention accuracy (0.948) but no measurable causal or evidential grounding.

arXiv:2608.15382v1 Announce Type: new Abstract: Large language models LLMs are increasingly proposed for healthcare decision support, but their evaluations still reward single-answer accuracy rather than reasoning about interventions, mechanisms, harms, evidence, and uncertainty. We propose a reproducible, graph-centered evaluation framework for intervention-oriented LLM behavior in healthcare and stress-test it in a cardiovascular pilot. The framework has four components: i a domain causal knowledge graph in which assertions are first-class, provenance-preserving nodes with stable identifiers; ii a scenario-conditioned subgraph extraction step that, given any clinical scenario, retrieves the relevant reified-assertion subgraph; iii four controlled grounding conditions that vary how the retrieved subgraph is composed into the model's context ungrounded C1, knowledge-graph C2, causal-graph C3, integrated C4 ; and iv an automated scoring pipeline, anchored on assertion identifiers, that computes intervention accuracy, and other evaluation measures on a single pass. To test the framework, we built a category-balanced scenario generator across eight reasoning failure modes and instantiated it on a cardiovascular graph. The metric panel discriminates conditions along interpretable, non-redundant axes: C4 obtains the strongest causal edge F1 0.838 , adverse-effect F1 0.833 , evidence accuracy 0.738 , and unsupported claim rate 0.114 , while C1 obtains the highest raw intervention accuracy 0.948 with no measurable causal or evidential grounding.