Joint architecture and knowledge distillation in CNN for Chinese text recognition. (March 2021)

• Record Type: Journal Article
• Authors: Wang, Zi-Rui; Du, Jun
• Is Part Of: Pattern recognition. Volume 111(2021)
• Publication Date: 2021-03
• Subjects: Convolutional neural network -- Acceleration and compression -- Architecture and knowledge distillation -- Offline handwritten Chinese text recognition; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2020.107722
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose a guideline to distill the architecture and knowledge of pre-trained standard CNNs simultaneously for fast compression and acceleration. The effectiveness is first verified on offline HCTR. Compared with the baseline CNN, the corresponding compact network can reduce the computational cost by > 10 × and model size by > 8 × with negligible accuracy loss. Furthermore, the proposed method is successfully used to reduce resource consumption of the mainstream backbone networks on CTW and MNIST. Abstract: The distillation technique helps transform cumbersome neural networks into compact networks so that models can be deployed on alternative hardware devices. The main advantage of distillation-based approaches include a simple training process, supported by most off-the-shelf deep learning software and no special hardware requirements. In this paper, we propose a guideline for distilling the architecture and knowledge of pretrained standard CNNs. The proposed algorithm is first verified on a large-scale task: offline handwritten Chinese text recognition (HCTR). Compared with the CNN in the state-of-the-art system, the reconstructed compact CNN can reduce the computational cost by > 10 × and the model size by > 8 × with negligible accuracy loss. Then, by conducting experiments on two additional classification task datasets: Chinese Text in the Wild (CTW) and MNIST, we demonstrate that the proposed approach can also be successfully applied on mainstream backbone networks.