Training much deeper spiking neural networks with a small number of time-steps. (September 2022)

• Record Type: Journal Article
• Authors: Meng, Qingyan; Yan, Shen; Xiao, Mingqing; Wang, Yisen; Lin, Zhouchen; Luo, Zhi-Quan
• Is Part Of: Neural networks. Volume 153(2022)
• Page Start: 254
• Page End: 268
• Publication Date: 2022-09
• Subjects: Spiking neural networks -- ANN-to-SNN conversion -- Conversion error analysis; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2022.06.001
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: Spiking Neural Network (SNN) is a promising energy-efficient neural architecture when implemented on neuromorphic hardware. The Artificial Neural Network (ANN) to SNN conversion method, which is the most effective SNN training method, has successfully converted moderately deep ANNs to SNNs with satisfactory performance. However, this method requires a large number of time-steps, which hurts the energy efficiency of SNNs. How to effectively covert a very deep ANN (e.g., more than 100 layers) to an SNN with a small number of time-steps remains a difficult task. To tackle this challenge, this paper makes the first attempt to propose a novel error analysis framework that takes both the "quantization error" and the "deviation error" into account, which comes from the discretization of SNN dynamicsthe neuron's coding scheme and the inconstant input currents at intermediate layers, respectively. Particularly, our theories reveal that the "deviation error" depends on both the spike threshold and the input variance. Based on our theoretical analysis, we further propose the Threshold Tuning and Residual Block Restructuring (TTRBR) method that can convert very deep ANNs (>100 layers) to SNNs with negligible accuracy degradation while requiring only a small number of time-steps. With very deep networks, our TTRBR method achieves state-of-the-art (SOTA) performance on the CIFAR-10, CIFAR-100, and ImageNet classification tasks. Highlights: Novel error decomposition for ANN-to-SNN conversion. Inconstant input current significantly hurts ANN-to-SNN conversion performance. Design threshold and network structure to mitigate the impact of inconstant current. Effectively convert 110-layer ResNets with low latency.