Hybrid Model Compression for Multi-Task Network. Issue 1 (1st August 2022)

• Record Type: Journal Article
• Authors: Xie, Min; Sun, Weize; Huang, Lei
• Is Part Of: Journal of physics. Volume 2320:Issue 1(2022)
• Publication Date: 2022-08-01
• Subjects: Physics -- Congresses; 530.5
• DOI: 10.1088/1742-6596/2320/1/012031
• Languages: English
• ISSNs: 1742-6588

Abstract

Abstract: In recent years, deep neural networks have made significant achievements in various fields in science and engineering. To achieve outstanding performance, neural network models with very deep structures and an extensive number of parameters are usually used, leading to huge memory storage and computational resources requirements of the hardware systems, and thus preventing them from being deployed in devices with hardware limits. In particular, such problem will be further amplified when multiple neural network models for several tasks are to be integrated into one single platform at the same time. In this paper, a hybrid joint-network optimization model is proposed to solve this multi-task multi-model compression problem. The idea of hard parameter sharing in multi-task learning is adopted. Specifically, we turn the sub-network of the same structure of multiple models into a hybrid network model, in which the same parameters are shared for all the tasks. In addition, a unique sparse matrix for each task is added to each task independently. Experimental results show that the proposed compression methods can not only achieve a high compression ratio but also get performance improvement for some tasks.