A generic shift-norm-activation approach for deep learning. (January 2021)

• Record Type: Journal Article
• Authors: Chen, Zhi; Ho, Pin-Han
• Is Part Of: Pattern recognition. Volume 109(2021)
• Publication Date: 2021-01
• Subjects: Activation -- Normalization -- CNN -- Shifting -- Deep learning; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2020.107609
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We present a shift-norm-activation framework, which integrates both normalization and activation into a single module, by filtering out signals with the found optimal threshold at both global scale and local scale. A rigorous mathematical analysis is performed on the understanding of normalization and activation on computation burden, comparisons to existing methods, its theoretical performance potential, and a reparameterization trick to improve optimization. The extensively conducted experiments demonstrate the potential of the proposed framework in various computer vision benchmarking tasks. Abstract: Deep learning has received increasing attention in the last decade. Its amazing success, is partly attributed to the evolution of normalization and activation techniques. However, less works have devoted to explore both modules together. This work, therefore, aims at pushing for a deeper understanding on the effect of normalization and activation together analytically. We design a generic method which integrates both normalization and activation together as a whole, named as the Generic Shift-Normalization-Activation Approach (GSNA), in reserving richer information propagation in neural networks. A rigorous mathematical analysis was performed to investigate the benefits of the designed method, such as its computation complexity, performance potential as well as optimization over trainable parameter initialization. Further, extensive experiments are conducted to demonstrate the superiority and generality of the designed method in many computer vision benchmarking tasks, such as CIFAR-10/100, SVHN, ImageNet32 × 32, etc. To explore its generality, we also conduct some experiments on natural language understanding tasks like text classification, natural language inference, and some variational generative task as well. More interestingly, GSNA can be naturally incorporated into the existing neural networks with arbitrary architectures, demonstrating its generic effectiveness in deep learning field.