Initializing photonic feed-forward neural networks using auxiliary tasks. (September 2020)

• Record Type: Journal Article
• Authors: Passalis, Nikolaos; Mourgias-Alexandris, George; Pleros, Nikos; Tefas, Anastasios
• Is Part Of: Neural networks. Volume 129(2020)
• Page Start: 103
• Page End: 108
• Publication Date: 2020-09
• Subjects: Photonic deep learning -- Neural network initialization -- Photonic activation functions; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2020.05.024
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: Photonics is among the most promising emerging technologies for providing fast and energy-efficient Deep Learning (DL) implementations. Despite their advantages, these photonic DL accelerators also come with certain important limitations. For example, the majority of existing photonic accelerators do not currently support many of the activation functions that are commonly used in DL, such as the ReLU activation function. Instead, sinusoidal and sigmoidal nonlinearities are usually employed, rendering the training process unstable and difficult to tune, mainly due to vanishing gradient phenomena. Thus, photonic DL models usually require carefully fine-tuning all their training hyper-parameters in order to ensure that the training process will proceed smoothly. Despite the recent advances in initialization schemes, as well as in optimization algorithms, training photonic DL models is still especially challenging. To overcome these limitations, we propose a novel adaptive initialization method that employs auxiliary tasks to estimate the optimal initialization variance for each layer of a network. The effectiveness of the proposed approach is demonstrated using two different datasets, as well as two recently proposed photonic activation functions and three different initialization methods. Apart from significantly increasing the stability of the training process, the proposed method can be directly used with any photonic activation function, without further requiring any other kind of fine-tuning, as also demonstrated through the conducted experiments. Highlights: Photonic neuromorphics is a highly promising emerging technology for accelerating DL. Significant architectural changes are required for photonic-based Deep Learning. An efficient initialization method for photonic Deep Learning models is proposed. The proposed method can efficiently work with different photonic activations. The effectiveness of the proposed method is validated on different computer vision tasks.