On the second-order optimization for spiking neural networks

Researchers proposed SpiKFAX, a second-order optimization method that builds a computationally tractable, Kronecker-factored approximation of the Fisher information matrix specifically adapted to spiking neural networks (SNNs), according to an arXiv paper (arXiv:2609.29379v1). The authors report that across five architectures and seven datasets, SpiKFAX consistently improved test accuracy and training stability compared with other popular optimizers, addressing the sharp loss landscape and sparse, discrete, temporally recurrent dynamics that complicate SNN training.

arXiv:2609.29379v1 Announce Type: new Abstract: Spiking Neural Networks SNNs offer an energy-efficient alternative to conventional neural networks by exploiting sparse, binary spikes, and event-driven computation. However, the training of SNNs remains challenging, as spiking activations create a sharp loss landscape that hinders training, and diagonal-curvature optimizers such as the Adam family may fail to capture this geometry. The extension of curvature-based optimization methods to SNNs is further complicated by the sparse, discrete, and temporally recurrent nature of their underlying dynamics. To address these limitations, we propose SpiKFAX, a second-order optimization method that formulates a computationally tractable, Kronecker-factored approximation of the Fisher information matrix specifically adapted to the structure of SNNs. Empirical evaluation across five architectures and seven datasets demonstrates that SpiKFAX consistently yields improvements in test accuracy and training stability relative to other popular optimizers.