Fluctuation-driven initialization for spiking neural network training. Issue 4 (1st December 2022)

• Record Type: Journal Article
• Authors: Rossbroich, Julian; Gygax, Julia; Zenke, Friedemann
• Is Part Of: Neuromorphic computing and engineering. Volume 2:Issue 4(2022)
• Publication Date: 2022-12-01
• Subjects: spiking neural networks -- surrogate gradients -- initialization strategies -- recurrent neural networks -- Dale's law -- homeostatic plasticity -- balanced networks; Neural networks (Computer science) -- Periodicals; Neural computers -- Periodicals; Neuromorphics -- Periodicals; 006.3
• DOI: 10.1088/2634-4386/ac97bb
• Languages: English
• ISSNs: 2634-4386

Abstract

Abstract: Spiking neural networks (SNNs) underlie low-power, fault-tolerant information processing in the brain and could constitute a power-efficient alternative to conventional deep neural networks when implemented on suitable neuromorphic hardware accelerators. However, instantiating SNNs that solve complex computational tasks in-silico remains a significant challenge. Surrogate gradient (SG) techniques have emerged as a standard solution for training SNNs end-to-end. Still, their success depends on synaptic weight initialization, similar to conventional artificial neural networks (ANNs). Yet, unlike in the case of ANNs, it remains elusive what constitutes a good initial state for an SNN. Here, we develop a general initialization strategy for SNNs inspired by the fluctuation-driven regime commonly observed in the brain. Specifically, we derive practical solutions for data-dependent weight initialization that ensure fluctuation-driven firing in the widely used leaky integrate-and-fire neurons. We empirically show that SNNs initialized following our strategy exhibit superior learning performance when trained with SGs. These findings generalize across several datasets and SNN architectures, including fully connected, deep convolutional, recurrent, and more biologically plausible SNNs obeying Dale's law. Thus fluctuation-driven initialization provides a practical, versatile, and easy-to-implement strategy for improving SNN training performance on diverse tasks in neuromorphic engineering and computational neuroscience.