Inside the LSTM: An XAI Field Guide to Weather Prediction

A developer demonstrates how to train an LSTM on daily temperature data and apply three explainability methods—permutation importance, SHAP, and Integrated Gradients—to interpret the model's predictions. The tutorial covers data preparation, model architecture with stacked LSTM layers and dropout, and early stopping, then evaluates which lag days the model relies on most.

LSTMs are still the go-to architecture for a lot of time series work, but they're annoying to trust. You get a number out the other end and no real sense of why the model landed there. This tutorial walks through training an LSTM on daily temperature data, then pulling it apart with three explainability methods: permutation importance, SHAP, and Integrated Gradients. Who this is for: people who already know some Keras and want to add interpretability to a forecasting model, not a from-scratch intro to neural nets. LSTMs want a 3D tensor — samples, timesteps, features — so before anything else we need to turn a flat column of temperatures into overlapping 7-day windows, each one paired with the value on day 8. python import numpy as np import pandas as pd from sklearn.preprocessing import MinMaxScaler 1. Load data df = pd.read csv "weather data.csv" data = df 'Temperature' .values.reshape -1, 1 2. Scale the data for stable neural network training scaler = MinMaxScaler feature range= 0, 1 scaled data = scaler.fit transform data 3. Create sequences: 7 days of lag to predict the 8th day X, y = , for i in range 7, len scaled data : X.append scaled data i-7:i y.append scaled data i X, y = np.array X , np.array y print f"Input shape: {X.shape}" Output: Samples, 7, 1 Scaling matters more than it sounds like it should — LSTMs trained on unscaled temperature values are prone to exploding gradients, and training just falls apart. The windowing step is really the whole trick here: every prediction only ever sees the past seven days, nothing more. Two stacked LSTM layers, dropout after each one, early stopping so we don't have to babysit the epoch count. python from tensorflow.keras.models import Sequential from tensorflow.keras.layers import LSTM, Dense, Dropout, Input from tensorflow.keras.callbacks import EarlyStopping 1. Build the network model = Sequential Input shape= 7, 1 , LSTM units=100, activation='relu', return sequences=True , Dropout 0.2 , LSTM units=100, activation='relu' , Dropout 0.2 , Dense units=1 model.compile optimizer='adam', loss='mse' 2. Configure Early Stopping early stopping = EarlyStopping monitor='val loss', patience=10, restore best weights=True 3. Train the model history = model.fit X, y, epochs=50, batch size=32, validation split=0.2, callbacks= early stopping One detail worth flagging: return sequences=True on the first LSTM layer is required, not optional, if you're stacking a second one — it passes the full sequence of hidden states forward instead of just the final one. patience=10 gives the model some room to wander before we give up on it, and restore best weights=True means we walk away with the checkpoint from its best epoch, not whatever it happened to end on. Once training is done, the obvious question is: does the model lean on yesterday's temperature, or does it weight the whole week roughly evenly? Three different methods, three different angles on that question. The idea is blunt but effective — shuffle one lag day across the whole dataset, see how much worse the predictions get, repeat for each of the 7 days. python from sklearn.metrics import mean squared error base preds = model.predict X, verbose=0 base error = mean squared error y, base preds feature importance = for i in range 7 : X permuted = X.copy np.random.shuffle X permuted :, i, 0 Shuffle a specific lag day permuted preds = model.predict X permuted, verbose=0 permuted error = mean squared error y, permuted preds feature importance.append permuted error - base error Whichever day causes the biggest jump in error when scrambled is the one the model depends on most. In practice, for a weather series, that's usually the most recent day or two — but it's worth checking rather than assuming. Permutation importance tells you what matters on average across the dataset. SHAP tells you what mattered for one specific prediction, which is a lot more useful when you're trying to explain a single forecast to someone. KernelExplainer works reasonably well for LSTMs, though be warned it's slow — it's model-agnostic and treats the network as a black box, so it has to make a lot of prediction calls. python import shap 1. Reshape data for KernelExplainer requires 2D input instead of 3D background data = X :100 .reshape -1, 7 instance to explain = X 0:1 .reshape -1, 7 2. Wrapper to handle 2D to 3D reshaping during SHAP calculation def model predict wrapper input 2d : return model.predict input 2d.reshape -1, 7, 1 , verbose=0 3. Compute SHAP Values explainer = shap.KernelExplainer model predict wrapper, background data shap values = explainer.shap values instance to explain 4. Visualize shap.initjs shap.force plot explainer.expected value 0 , shap values 0 .flatten , feature names= f'Day {i+1}' for i in range 7 , features=instance to explain 0,: The force plot shows, day by day, whether each lag pushed that particular forecast above or below the baseline average. It's the kind of chart you can actually put in front of a non-technical stakeholder. This one's gradient-based rather than perturbation-based, and it plays nicely with TensorFlow since it just needs access to the gradient tape. You interpolate a path from a baseline usually all zeros to the real input, and integrate the gradients along that path. python import tensorflow as tf def integrated gradients model, input tensor, baseline=None, steps=50 : if baseline is None: baseline = tf.zeros like input tensor 1. Interpolate from baseline to actual input alphas = tf.linspace 0.0, 1.0, steps + 1 interpolated inputs = baseline + alpha input tensor - baseline for alpha in alphas interpolated inputs = tf.concat interpolated inputs, axis=0 2. Calculate Gradients @tf.function def call model inputs : return model inputs, training=False with tf.GradientTape as tape: tape.watch interpolated inputs predictions = call model interpolated inputs grads = tape.gradient predictions, interpolated inputs 3. Approximate the integral avg grads = grads :-1 + grads 1: / 2.0 return tf.reduce mean avg grads, axis=0 input tensor - baseline sample = tf.convert to tensor X 0:1 , dtype=tf.float32 ig attributions = integrated gradients model, sample The steps parameter is a tradeoff — more steps gives a smoother, more accurate integral approximation, at the cost of more forward passes. 50 is a reasonable default; you rarely need to go much higher for a 7-day window. None of these three methods is strictly "correct" — they answer slightly different questions. Permutation importance gives you the global picture, SHAP gives you a per-prediction breakdown that's easy to explain to someone else, and Integrated Gradients gives you a gradient-based view that's cheap to compute since it doesn't need to touch the model as a black box. Running all three and comparing where they agree and where they don't tends to be more informative than picking just one.