{"slug": "the-explanation-gap-why-explainable-ai-still-struggles-to-speak-human", "title": "The Explanation Gap: Why Explainable AI Still Struggles to Speak Human", "summary": "A developer argues that explainable AI's core unresolved problem is not computing explanations but communicating them to the people who must act on them, without overstating their certainty. The piece traces the field from DARPA's 2015 XAI program through methods like LIME, counterfactual explanations, saliency maps, and SHAP, noting that raw accuracy metrics and technical outputs such as SHAP values fail to answer why a specific decision should be trusted. It cites the European Data Protection Supervisor's 2023 TechDispatch warning that opaque 'black box' decisions are unacceptable in high-stakes domains like healthcare, criminal justice, credit, and hiring.", "body_md": "A model predicts a 92% risk of hospital readmission. The SHAP values show age, prior admissions, and medication adherence as the top contributors. But when a nurse asks, \"Why *this* patient, *this* week?\" The technical answer: 'coefficients multiplied by feature values, printed to four decimal places'. Well this is correct as it is but almost useless in this conversation.\n\nThis is the unresolved core problem of explainable AI: getting a model to compute an answer was never the hard part in the long run. **Getting it to explain that answer to the person who actually has to act on it, without making the explanation sound more certain than it deserves to be**, is the part the field is still working through.\n\nIn 2015, DARPA launched its Explainable Artificial Intelligence (XAI) program with an explicit goal: to enable end users to \"understand, appropriately trust, and effectively manage\" AI systems. The program's retrospective makes clear that this was not an academic exercise. High-stakes domains (healthcare, criminal justice, credit, hiring) cannot tolerate decisions that are accurate on aggregate but inscrutable in individual cases.\n\nThe field has since developed a rich landscape of explanation methods. LIME (Local Interpretable Model-agnostic Explanations) fits simple surrogate models around individual predictions. **Counterfactual explanations** answer \"what would need to change for the outcome to flip?\". **Saliency** maps highlight influential pixels in images. **Attention-based explanations** point to tokens a transformer model \"attended to.\" **Rule extraction** methods distill complex models into human-readable if-then statements.\n\nNone of these methods, on their own, answers the question that matters most in high-stakes decisions: *Why should I, as a clinician or regulator or affected individual, believe this explanation?*\n\nRaw accuracy metrics \"the model is 92% accurate\" do not address this. They describe aggregate performance, not individual reasoning. A model can be highly accurate and still rely on spurious correlations, proxy variables for protected attributes, or patterns that will not hold under intervention. In healthcare, where a single false negative can cost a life, the \"why\" is not optional. It is the entire point.\n\nThe European Data Protection Supervisor's 2023 TechDispatch on XAI puts it plainly: \"It is therefore unacceptable to have a 'black box' effect that hides the underlying logic of decisions made by AI\". Opacity can hide bias, inaccuracies, and hallucinations. It prevents affected individuals from understanding, challenging, or correcting decisions that shape their lives.\n\nTo understand why the communication problem is so hard, you need to understand what SHAP values actually represent.\n\n**SHAP**(SHapley Additive exPlanations) is a method for attributing a model's prediction for a single instance to its input features. It is grounded in Shapley values from cooperative game theory, which answer: given a payoff produced by a coalition of players, how much credit does each player deserve?\n\nIn the ML context:\n\nFormally, for a feature *i* and a set of features *F*, the Shapley value is:\n\nwhere *f(S)* is the model's prediction when only features in *S* are known.\n\nI know, I know. The formula looks intimidating right? My bad. Lemme explain:\n\nImagine you're trying to figure out why your friend got 10/10 on a test.\n\nYou know they studied three things:\n\n\"How much did each of these things help my friend get that 10/10?\"\n\nBut there's a catch: studying might help a lot when combined with practice, but not as much without it.\n\nSo SHAP tries different combinations:\n\nStudying alone → score goes up a little\n\nStudying + practice → score goes up a lot\n\nStudying + sleep + practice → score goes up even more\n\nIt looks at all these different combinations and works out the average contribution of studying.\n\nThat's what a **Shapley value** is:\n\n**a way of fairly figuring out how much each feature helped produce the final prediction**.\n\nSHAP satisfies three desirable properties: **local accuracy** (the attributions sum to the prediction), **missingness** (features not present get zero attribution), and **consistency** (if a feature's contribution increases, its attribution does not decrease).\n\n**What SHAP is not**: a causal explanation.\n\nThis distinction is easy to lose the moment attributions get turned into sentences. A SHAP value of +0.18 for \"prior admissions\" means: *knowing* this patient's prior admissions pushes the model's prediction 18 percentage points above the baseline. It does not mean that *reducing* prior admissions would lower risk by 18 points in the real world.\n\nAs the SHAP documentation warns: \"SHAP makes transparent the correlations picked up by predictive ML models. But making correlations transparent does not make them causal!\". If the model has learned that zip code is a proxy for race in a biased dataset, SHAP will dutifully explain that spurious association.\n\nThis is not a bug in SHAP. It is a feature of any method that explains *model behavior* rather than *data-generating processes*. The moment you translate SHAP values into prose; \"this factor drove the prediction,\" \"this variable caused the risk\"; you risk smuggling in causal claims the method does not support.\n\nHere is the crux of the problem.\n\nA typical SHAP output for a single prediction looks like this (simplified):\n\n```\nBase value: 0.12\nshap_values: [0.08, -0.03, 0.15, 0.02, -0.01]\nfeature_names: ['age', 'income', 'prior_admissions', 'medication_adherence', 'distance_to_clinic']\n```\n\nOr, in a more structured form:\n\n```\nExplanation(\n    base_values=0.12,\n    values=[0.08, -0.03, 0.15, 0.02, -0.01],\n    feature_names=['age', 'income', 'prior_admissions', 'medication_adherence', 'distance_to_clinic']\n)\n```\n\nFor an ML engineer, this is informative as he/she reads Age: +0.08, Income: -0.03 fluently. For a nurse, a patient, a regulator, or a loan officer, it is nearly useless. The numbers live on a transformed scale (often log-odds). The feature names are internal identifiers, not clinical or business terms. A patient, a case worker, or a manager generally cannot, and wouldn't know how much weight to give it even if they could parse the syntax. There is no context: is 0.15 \"large\" for prior admissions, or typical?\n\nChristoph Molnar's *Interpretable Machine Learning* emphasizes this repeatedly: interpretability is not just about producing an explanation, but producing one that the *intended audience* can comprehend and use. A data scientist needs different information than a patient. A regulator needs different information than a clinician.\n\nThe EDPS TechDispatch puts it sharply: explanations should be \"presented in an understandable way, avoiding jargon and technical complexity\". Yet most XAI tools stop at the attribution layer.\n\nThere's a second, more consequential gap hiding underneath the first one: **an attribution describes how a model behaves, not how the world works**. A feature with a large positive SHAP value pushed this model's prediction upward; it did not \"cause\" the outcome in any scientific sense. A model can pick up a spurious correlation or a genuine but non-causal association, and the attribution method has no way to distinguish these, because it's a property of the model, not of reality. \n\nThis is the trap waiting for any tool that tries to translate attributions into plain sentences: it's easy, almost automatic, for \"this feature moved the prediction\" to slide into \"this feature causes the outcome\" once it's phrased as an English sentence rather than a number.\n\nSo the real bottleneck in xAI right now isn't producing more attributions, rather, building a responsible bridge from attribution to language: one that makes the numbers **legible** without making them **sound more certain**, or **more casual**, than they are. This is the gap that narrashap attempts to bridge.\n\nnarrashap, a small open-source Python library, is one working example of an attempt at that bridge; not a solved version of the problem, but a useful, inspectable case study in what building toward it actually requires. Its interface is a single function:\n\n``` python\nfrom narrashap import narrate\n\nnarrated_shap = narrate(\n    shap_values=shap_values,\n    instance=patient_row,\n    training_data=X_train,\n    risk_percentage=65.6,\n    risk_level=\"MODERATE RISK\",\n)\n\nprint(narrated_shap)\n```\n\nIt takes SHAP output for one prediction, the specific feature values for that case, and the training data the model learned from, and produces a narrative; either from a fixed set of plain-language sentence templates, or through a language model, depending on configuration. Consider the difference directly:\n\n```\nBase value: -0.114 (log-odds)\nPredicted value: 0.645 (log-odds)\nRace (Black): +1.58\nFamily history: -0.57\nVitamin D deficiency: -0.30\nHypertension: -0.13\nParity: +0.10\n```\n\n**Plain-language narrative (illustrative, not from a real tool):**\n\nThis patient's predicted risk of readmission is higher than average for our population. The model's prediction is driven primarily by their history of prior hospital admissions, which is in the 92nd percentile compared to similar patients. Age also contributes positively to the predicted risk. Income and medication adherence have smaller, offsetting effects, with higher income and good adherence slightly reducing the predicted risk. Distance to the clinic has minimal influence on this prediction.\n\n**Important:** This explanation describes how the model arrived at its prediction. It does not prove that these factors *cause* readmission risk, nor should it be used as a diagnosis or treatment recommendation.\n\nAsking an LLM nicely not to imply causation isn't sufficient on its own — it will occasionally slip. narrashap checks generated text against a list of banned causal phrases after generation, and rewrites if one appears. This has to be more careful than simple keyword matching: \"this is not proven to cause the outcome\" contains the word \"proven\" but is making exactly the hedged claim the system wants to allow. Getting the negation handling right took more than one attempt; an early fixed-word-lookback approach produced false positives on precisely the safe disclaimer language it needed to permit. It's a small, unglamorous detail, and exactly the kind that determines whether a safety mechanism works in practice or only in a demo.\n\nThis is the part of the problem that gets the least attention across the broader explainability field. Most tooling evaluates whether an attribution method is faithful to the model. Very little evaluates whether a narrative built on top of that attribution is faithful to the attribution itself; whether it mentioned the features that actually mattered, got the direction right, or invented a claim the data doesn't support. narrashap's fidelity scorer is a modest, direct attempt at treating that as a measurable property rather than an assumption.\n\nA patient, a clinician, and a fraud analyst don't need identical phrasing, though they need the same underlying discipline about what can be claimed. Domain-specific configuration; implemented here for healthcare and, more provisionally, fraud; lets terminology and tone shift while keeping the hedging requirements fixed.\n\nNone of these are solved problems, in this project or in the field generally. But they're the right questions, and they generalize well beyond this one library.\n\nThe narrative does several things:\n\nBoth the broader project of AI explainability and this particular tool remain unfinished work. narrashap has one meaningfully validated real-world integration — a logistic regression health-risk model — and its design decisions have only actually been tested against that setting. Other model families should work with the underlying extraction logic in principle but haven't been confirmed. It explains individual predictions; it is not a model governance tool, a fairness audit, or a substitute for evaluating a model's behavior across a population, and no explanation-narration layer, however careful, closes that gap on its own.\n\nBringing a language model into the loop reintroduces familiar risks that the field as a whole hasn't fully solved: occasional hallucination, a tendency toward misplaced confidence in tone, and provider-side model updates that quietly change behavior. Guardrails like a banned-phrase check make failures more likely to be caught — they don't make an LLM path risk-free. And the negation-aware safety check here has a known edge case: a compound sentence with an unrelated negation earlier in it could, in principle, let an unrelated unsafe claim through uncaught. A narrow, acknowledged gap, not a hidden one.\n\nBeyond narrashap specifically, the field still struggles with:\n\nThe future of XAI is not \"more explanations.\" It is better explanations: faithful, appropriately uncertain, legible to their actual audience, and honest about what they are not. \n\nHere are genuine, thoughtful directions—none of which narrashap (or most tools) claim to have solved yet:\n\nThe narrative layer should be evaluated as its own research problem, separate from the attribution method. Metrics might include:\n\nRecent work on SHAPstories and LLM-based narratives is a start. But standardized benchmarks and evaluation protocols are still emerging.\n\nThe EDPS TechDispatch emphasizes that explanations should be tailored to their audience. A patient needs different information than a regulator. A clinician needs different information than a data scientist.\n\nFuture tools should make audience a first-class parameter, not an afterthought. This means:\n\nWe monitor model accuracy drift. We should also monitor explanation drift: does the same input yield meaningfully different narratives after model retraining or LLM updates?\n\nThis is not just a technical problem. It is a governance problem. If explanations change, affected individuals and auditors need to know.\n\nAggregate fairness audits are important. But individuals affected by specific decisions need to know: was this decision influenced by a proxy for a protected attribute?\n\nFuture XAI tools should surface fairness-relevant signals at the individual explanation level, not just in aggregate reports.\n\nHuman review is not a transitional phase. It is a permanent design requirement for high-stakes explanations. Tools should make review easy:\n\nNarrashap is not a finished solution to the explanation gap. It is one honest, practical contribution to a much larger, still-unsolved problem.\n\nIt acknowledges that SHAP values alone are not enough. It bakes in guardrails against causal misinterpretation. It offers a zero-dependency fallback for sensitive environments. It treats the narrative layer as something that can (and should) be scored and audited.\n\nThe future of XAI will require many such contributions: tools that are faithful to their underlying attributions, appropriately uncertain, legible to their actual audiences, and honest about what they are not (a diagnosis, a proof, a causal claim).\n\nThe explanation gap will not close overnight. But projects like narrashap show that the field is moving in the right direction: from \"we have explanations\" to \"we have explanations we can trust.\"\n\nBelow is Narrashap's github [link](https://github.com/daviewisdm/narrashap)\n\n*Further reading: Lundberg, S. & Lee, S.-I. (2017), \"[A Unified Approach to Interpreting Model Predictions\", NeurIPS. Molnar, C., Interpretable Machine Learning.](https://arxiv.org/abs/1705.07874)*", "url": "https://wpnews.pro/news/the-explanation-gap-why-explainable-ai-still-struggles-to-speak-human", "canonical_source": "https://dev.to/daviewisdm/the-explanation-gap-why-explainable-ai-still-struggles-to-speak-human-13j6", "published_at": "2026-09-17 18:22:59+00:00", "updated_at": "2026-09-17 18:53:02.822449+00:00", "lang": "en", "topics": ["ai-safety", "ai-ethics", "ai-policy", "machine-learning", "artificial-intelligence"], "entities": ["DARPA", "European Data Protection Supervisor", "SHAP", "LIME", "Explainable Artificial Intelligence (XAI) program"], "alternates": {"html": "https://wpnews.pro/news/the-explanation-gap-why-explainable-ai-still-struggles-to-speak-human", "markdown": "https://wpnews.pro/news/the-explanation-gap-why-explainable-ai-still-struggles-to-speak-human.md", "text": "https://wpnews.pro/news/the-explanation-gap-why-explainable-ai-still-struggles-to-speak-human.txt", "jsonld": "https://wpnews.pro/news/the-explanation-gap-why-explainable-ai-still-struggles-to-speak-human.jsonld"}}