Towards Automatic Model Compression via a Unified Two-Stage Framework. (August 2023)

• Record Type: Journal Article
• Authors: Chen, Weihan; Wang, Peisong; Cheng, Jian
• Is Part Of: Pattern recognition. Volume 140(2023)
• Publication Date: 2023-08
• Subjects: Deep neural networks -- Model compression -- Quantization -- Pruning; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2023.109527
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose a unified two-stage framework for automatic model compression. We propose Dynamic BN to predict the performance of each compression policy. We propose a solving algorithm to search for the compression ratio allocation. Abstract: Deep Neural Networks have become ubiquitous in various domains. Meanwhile, the problems of massive storage and computation costs have hindered the deployment of these models to real-world applications. This paper proposes a novel and unified two-stage framework for automatic model compression. To determine the compression ratio of each layer, we improve the optimization from two aspects. First, to predict the performance of each compression policy, we propose Dynamic BN, which improves the correlation significantly with little computation overhead. Second, to search for the compression ratio allocation, we propose an efficient and hyperparameter-free solving algorithm based on the proposed Hessian matrix approximation and Knapsack problem reformulation. Moreover, comprehensive experiments and analyses are conducted on the CIFAR-100&ImageNet datasets and various network architectures to demonstrate its performance advantages over existing model compression methods under the quantization-only, pruning-only, and pruning-quantization settings.