Singular Learning and Occam's Razor in Deep Monomial Networks

Researchers used polynomial algebra to study critical points in deep monomial networks, finding that for large activation degrees, criticality occurs at subnetworks where neurons are inactive or redundant. This provides a mathematical explanation for the implicit bias of deep neural networks toward simpler functions.

arXiv:2606.28464v1 Announce Type: new Abstract: In the optimization of neural networks, gradient dynamics are influenced by critical points that arise from the model's architecture. These critical points occur where the Jacobian of the model's parametrization is rank-deficient, and are the most pronounced singularities studied in Singular Learning Theory. We investigate such points in deep fully-connected networks with monomial activations via tools from polynomial algebra such as Mason's Theorem. We show that, for sufficiently large activation degree, criticality occurs precisely at subnetworks, i.e., at parameter configurations where some neurons are inactive or redundant. This offers a mathematical perspective on the implicit bias in deep neural networks, explaining the tendency of these models to converge toward simpler functions.