Training Compact DNNs with ℓ1/2 Regularization. (April 2023)

• Record Type: Journal Article
• Authors: Tang, Anda; Niu, Lingfeng; Miao, Jianyu; Zhang, Peng
• Is Part Of: Pattern recognition. Volume 136(2023)
• Publication Date: 2023-04
• Subjects: Deep neural networks -- Model compression -- ℓ1/2 Quasi-norm -- Non-Lipschitz regularization -- Sparse optimization; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2022.109206
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose a network compression model based on ℓ 1 / 2 regularization. To the best of our knowledge, it is the first work utilizing non-Lipschitz continuous regularization to compress DNNs. We strictly prove the correspondence between ℓ p ( 0 < p < 1 ) and Hyper-Laplacian prior. Based on this prior, we suggest utilizing ℓ 1 / 2, as the single regularizer, to sparsify the connections and neurons of the network simultaneously. We give a closed-form, threshold solution to the proximal operator of ℓ 1 / 2, and consequently design a stochastic proximal gradient algorithm to train the resulting model. We conduct experiments to validate the performance of the proposed method. The results demonstrate that our method outperforms benchmark methods in terms of accuracy, computation and memory costs. Abstract: Deep neural network(DNN) has achieved unprecedented success in many fields. However, its large model parameters which bring a great burden on storage and calculation hinder the development and application of DNNs. It is worthy of compressing the model to reduce the complexity of the DNN. Sparsity-inducing regularizer is one of the most common tools for compression. In this paper, we propose utilizing the ℓ 1 / 2 quasi-norm to zero out weights of neural networks and compressing the networks automatically during the learning process. To our knowledge, it is the first work applying the non-Lipschitz continuous regularizer for the compression of DNNs. The resulting sparse optimization problem is solved by stochastic proximal gradient algorithm. For further convenience of calculation, an approximation of the threshold-form solution to the proximal operator with ℓ 1 / 2 is given at the same time. Extensive experiments with various datasets and baselines demonstrate the advantages of our new method.