Projection based weight normalization: Efficient method for optimization on oblique manifold in DNNs. (September 2020)

• Record Type: Journal Article
• Authors: Huang, Lei; Liu, Xianglong; Qin, Jie; Zhu, Fan; Liu, Li; Shao, Ling
• Is Part Of: Pattern recognition. Volume 105(2020:Sep.)
• Publication Date: 2020-09
• Subjects: Deep learning -- Weight normalization -- Oblique manifold -- Image classification; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2020.107317
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose to constrain the incoming weights of each neuron to be unit norm to address ill conditioned problem in DNNs. Constrain ing weight s can be formulated as an optimization problem over the Oblique manifold. We propose a simple yet efficient method referred to as projection based weight normalization (PBWN) to solve the optimization problem. PBWN has the property of regularization and collaborates well with the commonly used batch normalization technique. Extensive experiments on several widely used image datasets show the consistent performance improvement over the baseline DNNs. Abstract: Optimizing deep neural networks (DNNs) often suffers from the ill-conditioned problem. We observe that the scaling based weight space symmetry (SBWSS) in rectified nonlinear network will cause this negative effect. Therefore, we propose to constrain the incoming weights of each neuron to be unit-norm, which is formulated as an optimization problem over the Oblique manifold. A simple yet efficient method referred to as projection based weight normalization (PBWN) is also developed to solve this problem. This proposed method has the property of regularization and collaborates well with the commonly used batch normalization technique. We conduct comprehensive experiments on several widely-used image datasets including CIFAR-10, CIFAR-100, SVHN and ImageNet for supervised learning over the state-of-the-art neural networks. The experimental results show that our method is able to improve the performance of different architectures consistently. We also apply our method to Ladder network for semi-supervised learning on permutation invariant MNIST dataset, and our method achievers the state-of-the-art methods: we obtain test errors as 2.52%, 1.06%, and 0.91% with only 20, 50, and 100 labeled samples, respectively.