A model predicts a 92% risk of hospital readmission. The SHAP values show age, prior admissions, and medication adherence as the top contributors. But when a nurse asks, "Why this patient, this week?" The technical answer: 'coefficients multiplied by feature values, printed to four decimal places'. Well this is correct as it is but almost useless in this conversation.
This is the unresolved core problem of explainable AI: getting a model to compute an answer was never the hard part in the long run. Getting it to explain that answer to the person who actually has to act on it, without making the explanation sound more certain than it deserves to be, is the part the field is still working through.
In 2015, DARPA launched its Explainable Artificial Intelligence (XAI) program with an explicit goal: to enable end users to "understand, appropriately trust, and effectively manage" AI systems. The program's retrospective makes clear that this was not an academic exercise. High-stakes domains (healthcare, criminal justice, credit, hiring) cannot tolerate decisions that are accurate on aggregate but inscrutable in individual cases.
The field has since developed a rich landscape of explanation methods. LIME (Local Interpretable Model-agnostic Explanations) fits simple surrogate models around individual predictions. Counterfactual explanations answer "what would need to change for the outcome to flip?". Saliency maps highlight influential pixels in images. Attention-based explanations point to tokens a transformer model "attended to." Rule extraction methods distill complex models into human-readable if-then statements.
None of these methods, on their own, answers the question that matters most in high-stakes decisions: Why should I, as a clinician or regulator or affected individual, believe this explanation?
Raw accuracy metrics "the model is 92% accurate" do not address this. They describe aggregate performance, not individual reasoning. A model can be highly accurate and still rely on spurious correlations, proxy variables for protected attributes, or patterns that will not hold under intervention. In healthcare, where a single false negative can cost a life, the "why" is not optional. It is the entire point.
The European Data Protection Supervisor's 2023 TechDispatch on XAI puts it plainly: "It is therefore unacceptable to have a 'black box' effect that hides the underlying logic of decisions made by AI". Opacity can hide bias, inaccuracies, and hallucinations. It prevents affected individuals from understanding, challenging, or correcting decisions that shape their lives.
To understand why the communication problem is so hard, you need to understand what SHAP values actually represent.
SHAP(SHapley Additive exPlanations) is a method for attributing a model's prediction for a single instance to its input features. It is grounded in Shapley values from cooperative game theory, which answer: given a payoff produced by a coalition of players, how much credit does each player deserve?
In the ML context:
Formally, for a feature i and a set of features F, the Shapley value is:
where f(S) is the model's prediction when only features in S are known.
I know, I know. The formula looks intimidating right? My bad. Lemme explain:
Imagine you're trying to figure out why your friend got 10/10 on a test.
You know they studied three things:
"How much did each of these things help my friend get that 10/10?"
But there's a catch: studying might help a lot when combined with practice, but not as much without it.
So SHAP tries different combinations:
Studying alone → score goes up a little
Studying + practice → score goes up a lot
Studying + sleep + practice → score goes up even more
It looks at all these different combinations and works out the average contribution of studying.
That's what a Shapley value is:
a way of fairly figuring out how much each feature helped produce the final prediction.
SHAP satisfies three desirable properties: local accuracy (the attributions sum to the prediction), missingness (features not present get zero attribution), and consistency (if a feature's contribution increases, its attribution does not decrease).
What SHAP is not: a causal explanation.
This distinction is easy to lose the moment attributions get turned into sentences. A SHAP value of +0.18 for "prior admissions" means: knowing this patient's prior admissions pushes the model's prediction 18 percentage points above the baseline. It does not mean that reducing prior admissions would lower risk by 18 points in the real world.
As the SHAP documentation warns: "SHAP makes transparent the correlations picked up by predictive ML models. But making correlations transparent does not make them causal!". If the model has learned that zip code is a proxy for race in a biased dataset, SHAP will dutifully explain that spurious association.
This is not a bug in SHAP. It is a feature of any method that explains model behavior rather than data-generating processes. The moment you translate SHAP values into prose; "this factor drove the prediction," "this variable caused the risk"; you risk smuggling in causal claims the method does not support.
Here is the crux of the problem.
A typical SHAP output for a single prediction looks like this (simplified):
Base value: 0.12
shap_values: [0.08, -0.03, 0.15, 0.02, -0.01]
feature_names: ['age', 'income', 'prior_admissions', 'medication_adherence', 'distance_to_clinic']
Or, in a more structured form:
Explanation(
base_values=0.12,
values=[0.08, -0.03, 0.15, 0.02, -0.01],
feature_names=['age', 'income', 'prior_admissions', 'medication_adherence', 'distance_to_clinic']
)
For an ML engineer, this is informative as he/she reads Age: +0.08, Income: -0.03 fluently. For a nurse, a patient, a regulator, or a loan officer, it is nearly useless. The numbers live on a transformed scale (often log-odds). The feature names are internal identifiers, not clinical or business terms. A patient, a case worker, or a manager generally cannot, and wouldn't know how much weight to give it even if they could parse the syntax. There is no context: is 0.15 "large" for prior admissions, or typical?
Christoph Molnar's Interpretable Machine Learning emphasizes this repeatedly: interpretability is not just about producing an explanation, but producing one that the intended audience can comprehend and use. A data scientist needs different information than a patient. A regulator needs different information than a clinician.
The EDPS TechDispatch puts it sharply: explanations should be "presented in an understandable way, avoiding jargon and technical complexity". Yet most XAI tools stop at the attribution layer.
There's a second, more consequential gap hiding underneath the first one: an attribution describes how a model behaves, not how the world works. A feature with a large positive SHAP value pushed this model's prediction upward; it did not "cause" the outcome in any scientific sense. A model can pick up a spurious correlation or a genuine but non-causal association, and the attribution method has no way to distinguish these, because it's a property of the model, not of reality.
This is the trap waiting for any tool that tries to translate attributions into plain sentences: it's easy, almost automatic, for "this feature moved the prediction" to slide into "this feature causes the outcome" once it's phrased as an English sentence rather than a number.
So the real bottleneck in xAI right now isn't producing more attributions, rather, building a responsible bridge from attribution to language: one that makes the numbers legible without making them sound more certain, or more casual, than they are. This is the gap that narrashap attempts to bridge.
narrashap, a small open-source Python library, is one working example of an attempt at that bridge; not a solved version of the problem, but a useful, inspectable case study in what building toward it actually requires. Its interface is a single function:
from narrashap import narrate
narrated_shap = narrate(
shap_values=shap_values,
instance=patient_row,
training_data=X_train,
risk_percentage=65.6,
risk_level="MODERATE RISK",
)
print(narrated_shap)
It takes SHAP output for one prediction, the specific feature values for that case, and the training data the model learned from, and produces a narrative; either from a fixed set of plain-language sentence templates, or through a language model, depending on configuration. Consider the difference directly:
Base value: -0.114 (log-odds)
Predicted value: 0.645 (log-odds)
Race (Black): +1.58
Family history: -0.57
Vitamin D deficiency: -0.30
Hypertension: -0.13
Parity: +0.10
Plain-language narrative (illustrative, not from a real tool):
This patient's predicted risk of readmission is higher than average for our population. The model's prediction is driven primarily by their history of prior hospital admissions, which is in the 92nd percentile compared to similar patients. Age also contributes positively to the predicted risk. Income and medication adherence have smaller, offsetting effects, with higher income and good adherence slightly reducing the predicted risk. Distance to the clinic has minimal influence on this prediction.
Important: This explanation describes how the model arrived at its prediction. It does not prove that these factors cause readmission risk, nor should it be used as a diagnosis or treatment recommendation.
Asking an LLM nicely not to imply causation isn't sufficient on its own — it will occasionally slip. narrashap checks generated text against a list of banned causal phrases after generation, and rewrites if one appears. This has to be more careful than simple keyword matching: "this is not proven to cause the outcome" contains the word "proven" but is making exactly the hedged claim the system wants to allow. Getting the negation handling right took more than one attempt; an early fixed-word-lookback approach produced false positives on precisely the safe disclaimer language it needed to permit. It's a small, unglamorous detail, and exactly the kind that determines whether a safety mechanism works in practice or only in a demo.
This is the part of the problem that gets the least attention across the broader explainability field. Most tooling evaluates whether an attribution method is faithful to the model. Very little evaluates whether a narrative built on top of that attribution is faithful to the attribution itself; whether it mentioned the features that actually mattered, got the direction right, or invented a claim the data doesn't support. narrashap's fidelity scorer is a modest, direct attempt at treating that as a measurable property rather than an assumption.
A patient, a clinician, and a fraud analyst don't need identical phrasing, though they need the same underlying discipline about what can be claimed. Domain-specific configuration; implemented here for healthcare and, more provisionally, fraud; lets terminology and tone shift while keeping the hedging requirements fixed.
None of these are solved problems, in this project or in the field generally. But they're the right questions, and they generalize well beyond this one library.
The narrative does several things:
Both the broader project of AI explainability and this particular tool remain unfinished work. narrashap has one meaningfully validated real-world integration — a logistic regression health-risk model — and its design decisions have only actually been tested against that setting. Other model families should work with the underlying extraction logic in principle but haven't been confirmed. It explains individual predictions; it is not a model governance tool, a fairness audit, or a substitute for evaluating a model's behavior across a population, and no explanation-narration layer, however careful, closes that gap on its own.
Bringing a language model into the loop reintroduces familiar risks that the field as a whole hasn't fully solved: occasional hallucination, a tendency toward misplaced confidence in tone, and provider-side model updates that quietly change behavior. Guardrails like a banned-phrase check make failures more likely to be caught — they don't make an LLM path risk-free. And the negation-aware safety check here has a known edge case: a compound sentence with an unrelated negation earlier in it could, in principle, let an unrelated unsafe claim through uncaught. A narrow, acknowledged gap, not a hidden one.
Beyond narrashap specifically, the field still struggles with:
The future of XAI is not "more explanations." It is better explanations: faithful, appropriately uncertain, legible to their actual audience, and honest about what they are not.
Here are genuine, thoughtful directions—none of which narrashap (or most tools) claim to have solved yet:
The narrative layer should be evaluated as its own research problem, separate from the attribution method. Metrics might include:
Recent work on SHAPstories and LLM-based narratives is a start. But standardized benchmarks and evaluation protocols are still emerging.
The EDPS TechDispatch emphasizes that explanations should be tailored to their audience. A patient needs different information than a regulator. A clinician needs different information than a data scientist.
Future tools should make audience a first-class parameter, not an afterthought. This means:
We monitor model accuracy drift. We should also monitor explanation drift: does the same input yield meaningfully different narratives after model retraining or LLM updates?
This is not just a technical problem. It is a governance problem. If explanations change, affected individuals and auditors need to know.
Aggregate fairness audits are important. But individuals affected by specific decisions need to know: was this decision influenced by a proxy for a protected attribute?
Future XAI tools should surface fairness-relevant signals at the individual explanation level, not just in aggregate reports.
Human review is not a transitional phase. It is a permanent design requirement for high-stakes explanations. Tools should make review easy:
Narrashap is not a finished solution to the explanation gap. It is one honest, practical contribution to a much larger, still-unsolved problem.
It acknowledges that SHAP values alone are not enough. It bakes in guardrails against causal misinterpretation. It offers a zero-dependency fallback for sensitive environments. It treats the narrative layer as something that can (and should) be scored and audited.
The future of XAI will require many such contributions: tools that are faithful to their underlying attributions, appropriately uncertain, legible to their actual audiences, and honest about what they are not (a diagnosis, a proof, a causal claim).
The explanation gap will not close overnight. But projects like narrashap show that the field is moving in the right direction: from "we have explanations" to "we have explanations we can trust."
Below is Narrashap's github link
Further reading: Lundberg, S. & Lee, S.-I. (2017), "A Unified Approach to Interpreting Model Predictions", NeurIPS. Molnar, C., Interpretable Machine Learning.