Adversarial co-distillation learning for image recognition. (March 2021)

• Record Type: Journal Article
• Authors: Zhang, Haoran; Hu, Zhenzhen; Qin, Wei; Xu, Mingliang; Wang, Meng
• Is Part Of: Pattern recognition. Volume 111(2021)
• Publication Date: 2021-03
• Subjects: Knowledge distillation -- Data augmentation -- Generative adversarial nets -- Divergent examples -- Image classification; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2020.107659
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: In the process of knowledge co-distillation, we find that some special images, e.g., the divergent examples, can improve the generalization performance of the deep neural networks, and these images are scarce. For assisting knowledge co-distillation, we design an end-to-end framework named Adversarial Co-distillation Networks (ACNs) to generate extra divergent examples. To improve the quality of the divergent examples, we develop Weakly Residual Connection and Restricted Adversarial Search. We conduct extensive experiments with various architectures and datasets. Abstract: Knowledge distillation is an effective way to transfer the knowledge from a pre-trained teacher model to a student model. Co-distillation, as an online variant of distillation, further accelerates the training process and paves a new way to explore the "dark knowledge" by training n models in parallel. In this paper, we explore the "divergent examples", which can make the classifiers have different predictions and thus induce the "dark knowledge", and we propose a novel approach named Adversarial Co-distillation Networks (ACNs) to enhance the "dark knowledge" by generating extra divergent examples. Note that we do not involve any extra dataset, and we only utilize the standard training set to train the entire framework. ACNs are end-to-end frameworks composed of two parts: an adversarial phase consisting of Generative Adversarial Networks (GANs) to generate the divergent examples and a co-distillation phase consisting of multiple classifiers to learn the divergent examples. These two phases are learned in an iterative and adversarial way. To guarantee the quality of the divergent examples and the stability of ACNs, we further design "Weakly Residual Connection" module and "Restricted Adversarial Search" module to assist in the training process. Extensive experiments with various deep architectures on different datasets well demonstrate the effectiveness of our approach.