Synaptic 1/f noise injection for overfitting suppression in hardware neural networks. Issue 3 (1st September 2022)

• Record Type: Journal Article
• Authors: Du, Yan; Shao, Wei; Chai, Zheng; Zhao, Hanzhang; Diao, Qihui; Gao, Yawei; Yuan, Xihui; Wang, Qiaoqiao; Li, Tao; Zhang, Weidong; Zhang, Jian Fu; Min, Tai
• Is Part Of: Neuromorphic computing and engineering. Volume 2:Issue 3(2022)
• Publication Date: 2022-09-01
• Subjects: neuromorphic computing -- 1/f noise -- overfitting -- hardware neural network; Neural networks (Computer science) -- Periodicals; Neural computers -- Periodicals; Neuromorphics -- Periodicals; 006.3
• DOI: 10.1088/2634-4386/ac6d05
• Languages: English
• ISSNs: 2634-4386

Abstract

Abstract: Overfitting is a common and critical challenge for neural networks trained with limited dataset. The conventional solution is software-based regularization algorithms such as Gaussian noise injection. Semiconductor noise, such as 1/ f noise, in artificial neuron/synapse devices, which is often regarded as undesirable disturbance to the hardware neural networks (HNNs), could also play a useful role in suppressing overfitting, but that is as yet unexplored. In this work, we proposed the idea of using 1/ f noise injection to suppress overfitting in different neural networks, and demonstrated that: (i) 1/ f noise could suppress the overfitting in multilayer perceptron (MLP) and long short-term memory (LSTM); (ii) 1/ f noise and Gaussian noise performs similarly for the MLP but differently for the LSTM; (iii) the superior performance of 1/ f noise on LSTM can be attributed to its intrinsic long range dependence. This work reveals that 1/ f noise, which is common in semiconductor devices, can be a useful solution to suppress the overfitting in HNNs, and more importantly, further evidents that the imperfectness of semiconductor devices is a rich mine of solutions to boost the development of brain-inspired hardware technologies in the artificial intelligence era.