Relevance Propagation as Entropic Optimal Transport on Filippov Neural Networks: Sinkhorn Propagation and Region Relevance in Image Classifiers

A new arXiv paper (2611.05594v1) shows that Layer-wise Relevance Propagation (LRP) is not a heuristic but the solution of a single variational problem: the divergence-free flow on the active computational graph of zero relative entropy to the backward transition law of the forward activation flow. The authors formalize feed-forward networks as directed acyclic graphs of Filippov operators, show that LRP$_0$, LRP$_{z^+}$ and LRP$_{\alpha,\beta}$ are instances of one rule LRPN$_{\alpha,\beta}$, and derive a new rule, Sinkhorn Relevance Propagation, by taking entropy edge by edge. On VGG-16 the conservation identities hold to round-off, and on ConvNeXt-Tiny LRPN$_{\alpha,\beta}$ conserves the output while its Hebbian weights give the highest ABPC of the rules tested.

arXiv:2611.05594v1 Announce Type: new Abstract: Relevance propagation, which redistributes a network's prediction backward through the layers, is usually treated as a heuristic. We show that it is the solution of a single variational problem: Layer-wise Relevance Propagation LRP is the divergence-free flow on the active computational graph of zero relative entropy to the backward transition law of the forward activation flow; for a non-negative reference, this is a relative entropy between Markov measures on backward paths. The rules LRP$ 0$, LRP$ {z^+}$ and LRP$ {\alpha,\beta}$ follow from this principle by the choice of a reference, and are instances of one rule, LRPN$ {\alpha,\beta}$, whose weights may vary from node to node, for instance with the node's activity, in the spirit of Hebb's principle. The principle rests on a measure-theoretic formalization of feed-forward networks as directed acyclic graphs of Filippov operators, which covers piecewise-smooth and discontinuous architectures e.g. hard-routed Mixture-of-Experts with a Clarke-Filippov generalized-gradient calculus; it also characterizes backpropagation as the adjoint equation of a Lagrangian relaxation. On networks of ReLU and max-pooling nodes, LRP$ 0$ is Gradient$\times$Input, with an explicit jump across a kink where LRP$ {z^+}$ stays continuous. Taking the entropy edge by edge yields a new rule, Sinkhorn Relevance Propagation. Aggregated over an input region, the flow of a non-negative reference tells, with exact accounting, which hidden units carry its evidence, the conditioned flow being a Schr\"odinger bridge. On VGG-16, the conservation identities hold to round-off, and the rules are told apart by the sign of their maps, the share of the output left to the bias nodes, faithfulness, localization and class dependence; on ConvNeXt-Tiny, LRPN$ {\alpha,\beta}$ conserves the output and its Hebbian weights give the highest ABPC of the rules tested.