Puzzling Success of Overparameterization: Lottery Tickets or Escape Dimensions?

Researchers argue that the 'lottery ticket' analogy for overparameterized neural networks is misleading, proposing instead that wider networks succeed by expanding optimization dimensions to escape bad minima. The study highlights that winning tickets can fail when the rest of the network is perturbed, challenging the isolated subnetwork view.

The Puzzling Success of Overparameterization: Lottery Tickets or Escape Dimensions? Lotteries and tickets are often used as a didactical analogy to explain the success of overparameterized neural networks: “larger networks succeed because they more likely contain a well-initialized subnetwork that can learn the task in isolation, much like buying more tickets increases the chances of winning a lottery.” This explanation is intuitive but misleading: it suggests that subnetworks can be treated in isolation from the rest of the network. Following this reasoning leads to interpreting learning in wide networks as a multi-start optimization process, where gradient descent simply conducts a parallel search over subnetworks. We argue that this view is flawed since, among other reasons, winning tickets can be made to fail by perturbing the rest of the network. We put forward a more accurate intuitive picture for the success of overparameterization based on the geometry of loss landscapes: increasing width expands the set of available dimensions for optimization, making it easier to escape bad local minima. Moreover, as width grows, bad minima become increasingly rare relative to good minima. As the field grows mature, it is important to refine the analogies we use to explain foundational phenomena, such as the apparent redundancy of large networks, reconciling practitioners' intuitions with modern theoretical insights. Name EDT.pdf Type Main Document Version Submitted version Preprint Access type openaccess License Condition CC BY Size 7.87 MB Format Adobe PDF Checksum MD5 1153dd475c0b09ff6fd334269c37e63c