3 Ways to Enhance Your AI Model’s Interpretability

A new article outlines three techniques for enhancing AI model interpretability—SHAP, LIME, and Integrated Gradients—applied to a customer churn model, emphasizing the EU AI Act's Article 13 requirement for transparency in high-risk AI systems. The piece argues that traditional feature importance scores are insufficient for local explanations and can be biased toward high-cardinality features.

In this article, you will learn three concrete techniques for making machine learning model predictions interpretable, covering both global and local explanations across tree-based and neural network architectures. Topics we will cover include: - Why traditional feature importance scores fall short as a complete interpretability solution, and when they mislead. - How SHAP, LIME, and Integrated Gradients each work, and what makes each one suited to different deployment constraints. - How to apply all three techniques to the same customer churn example so their explanations can be directly compared. A model that predicts accurately and a model whose reasoning you can actually explain are two different achievements, and only one of them is optional anymore. A churn model that flags a loyal, five-year customer as high-risk isn’t just an interesting edge case if nobody on the team can say why; it’s a decision nobody can defend, to a manager, to the customer, or increasingly, to a regulator. The EU AI Act’s Article 13 https://artificialintelligenceact.eu/article/13/ now requires high-risk AI systems to provide sufficient transparency for deployers to actually interpret their outputs, which has moved interpretability from a nice-to-have research topic to a genuine deployment requirement for a growing share of real systems. This article covers three concrete, current techniques for getting real answers out of a model that would otherwise stay a black box. One example runs through the whole piece: a customer churn prediction model, first a gradient-boosted tree, later a small neural network trained on the same data, so every technique is explaining the same underlying problem rather than jumping between disconnected toy examples. What Model Interpretability Actually Means Model interpretability is the degree to which a human can understand why a model produced a specific output, not just that it produced one. That definition splits cleanly into two questions that get conflated constantly, and untangling them now saves confusion in every section after this one. Global interpretability asks how the model behaves overall: across the whole dataset, which features matter most, and in which direction. Local interpretability asks something narrower and, for most real decisions, more important: why did the model make this prediction, for this customer, right now? A model can be reasonably interpretable globally — “tenure and contract length matter most on average” — while still being a total mystery locally, since knowing what matters on average tells you nothing about why one specific loyal customer just got flagged as a churn risk. The Traditional Method, and Why It Doesn’t Scale Ask most data scientists how to explain a tree-based model and the first answer is usually the same: pull the built-in .feature importances attribute that ships with practically every scikit-learn ensemble model, or read the coefficients straight off a linear model. It’s fast, it requires no extra library, and it gives you a ranked list in one line of code. importances = pd.Series model.feature importances , index=FEATURES .sort values ascending=False 12 importances = pd.Series model.feature importances ,index=FEATURES .sort values ascending=False Run against the churn model, this returns tenure at the top, followed by monthly charge, support tickets, contract type, and late payments. That’s a real answer, and it’s also where the traditional method’s real limits start showing up. It’s global-only by construction; it can tell you tenure matters most across the whole customer base, but it says nothing at all about why one specific customer — someone with five years of tenure who should look safe — just got flagged as high-risk. It can also be measurably biased toward high-cardinality features https://www.sanfoundry.com/model-interpretability-shap-lime-feature-importance/ , inflating the apparent importance of a variable simply because it has more possible split points, not because it’s genuinely more predictive. And it only exists at all for models that happen to expose that attribute; the moment you’re working with something that doesn’t ship a built-in importance score — a neural network, an ensemble of mixed model types, a black-box API you’re calling — this method has nothing to offer. That gap — no per-prediction explanation, a bias baked into how the score is computed, and no coverage outside a narrow set of model types — is exactly what the three techniques below exist to close. Prerequisites - Python 3.11+ - pip install shap lime scikit-learn pandas numpy torch captum 1 pip install shap lime scikit-learn pandas numpy torch captum Every code snippet in the three sections below imports from one shared file, churn data.py , which builds the synthetic churn dataset and trains the gradient-boosted tree model used in Ways 1 and 2. Save this first, before running anything else: python churn data.py import numpy as np import pandas as pd from sklearn.ensemble import GradientBoostingClassifier from sklearn.model selection import train test split rng = np.random.default rng 42 n = 2000 tenure months = rng.integers 1, 72, n monthly charge = rng.normal 70, 25, n .clip 15, 200 support tickets = rng.poisson 1.5, n contract is monthly = rng.integers 0, 2, n 1 = month-to-month, 0 = annual+ late payments = rng.poisson 0.8, n True churn logic: short tenure, month-to-month contracts, and lots of support tickets all push churn probability up; long tenure pulls it down logit = -1.5 - 0.04 tenure months + 0.015 monthly charge + 0.35 support tickets + 1.1 contract is monthly + 0.25 late payments prob churn = 1 / 1 + np.exp -logit churned = rng.uniform 0, 1, n < prob churn .astype int df = pd.DataFrame { "tenure months": tenure months, "monthly charge": monthly charge, "support tickets": support tickets, "contract is monthly": contract is monthly, "late payments": late payments, "churned": churned, } FEATURES = "tenure months", "monthly charge", "support tickets", "contract is monthly", "late payments" X = df FEATURES y = df "churned" X train, X test, y train, y test = train test split X, y, test size=0.2, random state=42 model = GradientBoostingClassifier random state=42 model.fit X train, y train if name == " main ": print f"Train accuracy: {model.score X train, y train :.3f}" print f"Test accuracy: {model.score X test, y test :.3f}" print f"Churn rate in data: {y.mean :.1%}" 12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849 churn data.pyimport numpy as npimport pandas as pdfrom sklearn.ensemble import GradientBoostingClassifierfrom sklearn.model selection import train test split rng = np.random.default rng 42 n = 2000 tenure months = rng.integers 1, 72, n monthly charge = rng.normal 70, 25, n .clip 15, 200 support tickets = rng.poisson 1.5, n contract is monthly = rng.integers 0, 2, n 1 = month-to-month, 0 = annual+late payments = rng.poisson 0.8, n True churn logic: short tenure, month-to-month contracts, and lots of support tickets all push churn probability up; long tenure pulls it downlogit = -1.5 - 0.04 tenure months + 0.015 monthly charge + 0.35 support tickets + 1.1 contract is monthly + 0.25 late payments prob churn = 1 / 1 + np.exp -logit churned = rng.uniform 0, 1, n < prob churn .astype int df = pd.DataFrame { "tenure months": tenure months, "monthly charge": monthly charge, "support tickets": support tickets, "contract is monthly": contract is monthly, "late payments": late payments, "churned": churned,} FEATURES = "tenure months", "monthly charge", "support tickets", "contract is monthly", "late payments" X = df FEATURES y = df "churned" X train, X test, y train, y test = train test split X, y, test size=0.2, random state=42 model = GradientBoostingClassifier random state=42 model.fit X train, y train if name == " main ": print f"Train accuracy: {model.score X train, y train :.3f}" print f"Test accuracy: {model.score X test, y test :.3f}" print f"Churn rate in data: {y.mean :.1%}" What this does : the churn label isn’t random; it’s generated from a real logistic relationship where short tenure, a month-to-month contract, and a high support-ticket count all genuinely increase churn probability, with some random noise mixed in so the model doesn’t get a suspiciously perfect signal. That matters for this article specifically: every interpretability technique below is being tested against a dataset where the true underlying drivers of churn are actually known in advance, which is what makes it possible to judge whether each method’s explanation is plausible rather than just plausible-sounding. Run this file directly python churn data.py , and it reports a test accuracy of 0.698 against a 36.8% baseline churn rate — a real, moderately skilled model, not a toy that memorized the data. The customer referenced in the three sections below is X test.iloc 0 , the same specific customer, 53 months of tenure and 5 recent support tickets, used consistently across SHAP, LIME, and Integrated Gradients so their explanations can be compared directly. Method 1: SHAP SHapley Additive exPlanations SHAP is grounded in cooperative game theory: treat each feature as a player in a game where the model’s output is the payout, and compute each feature’s fair share of that payout by averaging its marginal contribution across every possible combination of features it could be considered alongside. That sounds abstract, but the practical result is a single, mathematically consistent method that produces both global and local explanations, unlike the traditional method, which only gave you one of those two. SHAP is currently at version 0.52.0, released May 28, 2026 https://pypi.org/project/shap/ , and remains the most widely adopted interpretability library in production use. python import shap import numpy as np import pandas as pd from churn data import model, X test, FEATURES explainer = shap.TreeExplainer model shap values = explainer X test Global: average absolute contribution per feature across every prediction mean abs = np.abs shap values.values .mean axis=0 global importance = pd.Series mean abs, index=FEATURES .sort values ascending=False 1234567891011 import shapimport numpy as npimport pandas as pdfrom churn data import model, X test, FEATURES explainer = shap.TreeExplainer model shap values = explainer X test Global: average absolute contribution per feature across every predictionmean abs = np.abs shap values.values .mean axis=0 global importance = pd.Series mean abs, index=FEATURES .sort values ascending=False Running this against the same churn model produces a genuinely different ranking than the traditional method did: contract is monthly jumps from fourth place under .feature importances to second place under SHAP, while support tickets drops from third to fourth. That’s not a rounding difference; it’s two different, both-reasonable methods disagreeing on how much a feature actually matters, and it’s exactly the kind of discrepancy that makes relying on a single crude score risky. The local explanation is where SHAP earns its keep, though. Pull the specific customer from the example above — someone with 53 months of tenure but 5 recent support tickets: customer shap = shap values.values 0 this customer's per-feature contribution 1 customer shap = shap values.values 0 this customer's per-feature contribution The result : support tickets contributes +2.81 to this customer’s churn log-odds, by far the largest single push toward churn, while tenure months pulls in the opposite direction at only -0.58. The two effects don’t cancel out. This customer’s long tenure, which looked protective in the global ranking, isn’t enough to outweigh a real support-ticket problem, and the model’s actual predicted probability lands at 89.5% churn risk. That’s a specific, defensible answer to “ why did the model flag this customer ,” not an average across thousands of customers who aren’t this one. SHAP’s real cost is computational. TreeSHAP https://shap.readthedocs.io/en/latest/generated/shap.TreeExplainer.html , the variant used here, is fast specifically because it exploits the structure of tree-based models directly, but the more general KernelSHAP variant needed for arbitrary model types requires far more model evaluations per explanation, which is the opening for the next technique. Method 2: LIME Local Interpretable Model-agnostic Explanations LIME takes a fundamentally different approach: rather than computing a game-theoretically exact attribution, it generates a cloud of perturbed samples around one specific prediction, weights them by proximity to the original input, and fits a simple, interpretable model — typically a linear one — on that local neighbourhood. The result approximates how the real model behaves right around this one prediction, without needing to understand anything about the real model’s internal structure. python import pandas as pd from lime.lime tabular import LimeTabularExplainer from churn data import model, X train, X test, FEATURES customer = X test.iloc 0 explainer = LimeTabularExplainer X train.values, feature names=FEATURES, class names= "stayed", "churned" , mode="classification", random state=42, def predict proba df x : return model.predict proba pd.DataFrame x, columns=FEATURES explanation = explainer.explain instance customer.values, predict proba df, num features=5 123456789101112131415 import pandas as pdfrom lime.lime tabular import LimeTabularExplainerfrom churn data import model, X train, X test, FEATURES customer = X test.iloc 0 explainer = LimeTabularExplainer X train.values, feature names=FEATURES, class names= "stayed", "churned" , mode="classification", random state=42, def predict proba df x : return model.predict proba pd.DataFrame x, columns=FEATURES explanation = explainer.explain instance customer.values, predict proba df, num features=5 Run against the same customer used in the SHAP example, LIME’s explanation lines up remarkably well: support tickets 2.00 contributes the largest positive weight toward churn, while contract is monthly <= 0.00 and the customer’s longer tenure bracket both pull the other way — the same story SHAP told, arrived at through a completely different mechanism. That agreement between two independently built methods is itself a useful signal; when SHAP and LIME diverge sharply on the same prediction, that’s usually worth investigating rather than picking whichever answer you like better. Where LIME genuinely wins is speed. It doesn’t need to reason about the model’s full structure or run the many evaluations SHAP’s more general variants require, which makes it the more practical choice when you’re explaining predictions inside a real-time system with a tight latency budget, or working with a model type SHAP doesn’t have a fast, specialized explainer for. The trade-off is real too: because LIME’s local surrogate depends on randomly sampled perturbations, running the exact same explanation twice can produce slightly different weights — a lack of stability SHAP’s game-theoretic foundation doesn’t share. Method 3: Integrated Gradients The first two techniques both treat the model as a black box, which is useful because it means they work on anything, but it also means they can’t take advantage of a model’s internal structure when that structure is actually available. Integrated Gradients is built specifically for differentiable models — such as neural networks — where you can walk a straight-line path from a neutral baseline input to the real one and accumulate the gradient of the output with respect to each feature along every step of that path. The accumulated gradient tells you how much each feature’s actual value, relative to the baseline, drove the final prediction. For this technique, the churn model must actually be a neural network, so a small one was trained on the identical dataset used above — same features, same customers, same train/test split — just a different model architecture entirely. python import torch from captum.attr import IntegratedGradients from churn data import X test, FEATURES Assumes net is a trained PyTorch model and customer normalized is the normalized feature vector for X test.iloc 0 net.eval input tensor = torch.tensor customer normalized, dtype=torch.float32 .unsqueeze 0 input tensor.requires grad baseline = torch.zeros like input tensor an "average" customer after normalization ig = IntegratedGradients net attributions, delta = ig.attribute input tensor, baseline, return convergence delta=True, n steps=200 12345678910111213 import torchfrom captum.attr import IntegratedGradientsfrom churn data import X test, FEATURES Assumes net is a trained PyTorch model and customer normalized is the normalized feature vector for X test.iloc 0 net.eval input tensor = torch.tensor customer normalized, dtype=torch.float32 .unsqueeze 0 input tensor.requires grad baseline = torch.zeros like input tensor an "average" customer after normalization ig = IntegratedGradients net attributions, delta = ig.attribute input tensor, baseline, return convergence delta=True, n steps=200 What this does : the baseline represents a neutral reference point — here, a customer at the average value for every feature, since the inputs were normalized before training. n steps controls how finely the path between baseline and real input gets sampled, and return convergence delta is a genuine sanity check worth using every time: it measures how closely the sum of the attributions matches the actual difference between the model’s output on the real input and on the baseline, and it should land close to zero if the computation is numerically sound. In this run, the convergence delta came back at 0.0006 — essentially zero — confirming the attribution is trustworthy rather than a noisy approximation. Run against the same customer profile as the SHAP and LIME examples, Integrated Gradients tells the same story a third time: support tickets produces the largest positive attribution by a wide margin, while tenure months and contract is monthly both pull toward “stay.” Three structurally different techniques — a game-theoretic attribution, a local linear surrogate, and a gradient-path integration — independently converging on the same explanation for the same customer is about as strong a confirmation as interpretability tooling can offer that the explanation reflects something real about the model’s behavior, not an artifact of any one method. Which One to Actually Reach For These three aren’t competing options where one is simply best; they’re suited to different constraints, and the honest answer depends on your model and your situation. Reach for SHAP when you’re working with tree-based models specifically where TreeSHAP is fast , and you want both a global picture and airtight local explanations from one consistent, theoretically grounded method. Reach for LIME when compute or latency is genuinely tight, or when you need a quick local explanation for a model type without a specialized fast SHAP variant, accepting that the explanation may shift slightly between runs. Reach for Integrated Gradients the moment your model is a neural network or otherwise differentiable, since it’s the only one of the three built to actually use that structure rather than treating the model as an opaque function. Conclusion The traditional feature-importance score isn’t wrong; it’s incomplete: a single global number that can’t explain one prediction, can’t be trusted uniformly across feature types, and doesn’t exist at all for a growing share of the models teams actually deploy. SHAP, LIME, and Integrated Gradients each close that gap differently, and picking one before a regulator, a confused customer, or your own team forces the question is the actual habit worth building. The churn example throughout this piece made that concrete: three different methods, three different mechanisms, and the same honest answer for the same customer — which is exactly what a model you can genuinely trust should look like under examination.