Lower expected Fisher rank implies better PAC generalization

A new arXiv paper (2610.33208v1) defines a Fisher-aligned parameter distribution for PAC-Bayes analysis that yields a high-probability generalization error bound scaling with the expected rank of the Fisher information matrix. The authors show the distribution simultaneously guarantees low loss, by constraining exploration to functional invariances, and a large log-volume of parameters proportional to the number of zero eigenvalues. The result links a lower expected Fisher rank to better PAC generalization, offering a theoretical explanation for why deep learning architectures generalize to unseen data under i.i.d. assumptions.

arXiv:2610.33208v1 Announce Type: new Abstract: Deep learning architectures tend to generalize well to unseen data, but explaining this capability is still an open area of research. Under the assumption of i.i.d. data, PAC-Bayes provides a bound on generalization error, and its tightness relies on a parameter distribution that covers a large volume of low-loss solutions. The Fisher information metric endows the parameter space with a geometry for which distances reflect functional changes, and whose zero-component captures functional invariances. We define and study a Fisher-aligned parameter distribution for which we can simultaneously guarantee 1 a low-loss, thanks to the exploration constrained on the functional invariances and 2 a large log-volume of parameters, proportional to the number of zero eigenvalues. This distribution results in a high-probability generalization error bound, which scales with the expected rank of the Fisher information matrix, shedding light on the relevance of this quantity.