Knowledge distillation-based deep learning classification network for peripheral blood leukocytes. (May 2022)

• Record Type: Journal Article
• Authors: Leng, Bing; Leng, Min; Ge, Mingfeng; Dong, Wenfei
• Is Part Of: Biomedical signal processing and control. Volume 75(2022)
• Publication Date: 2022-05
• Subjects: Knowledge distillation -- Deep learning -- Leukocyte classification -- CNN -- Transformer; Signal processing -- Periodicals; Biomedical engineering -- Periodicals; Signal Processing, Computer-Assisted -- Periodicals; Image Processing, Computer-Assisted -- Periodicals; Biomedical Engineering -- Periodicals; 610.28
• DOI: 10.1016/j.bspc.2022.103590
• Languages: English
• ISSNs: 1746-8094

Abstract

Highlights: Knowledge distillation can combine the advantages of ViT and CNN. ViT's classification performance is slightly worse than CNN on the leukocyte dataset. ViT guides the student CNN to achieve the highest performance, with or without ground truth. The category probability distribution after knowledge distillation of ViT is different from that of CNN. ViT can compensate for CNN's lack of ability to extract global features. Abstract: Leukocytes are a critical component of the human immune system. Many diseases can be diagnosed by analyzing the morphology and number of leukocytes. Due to the extensive application of convolutional neural networks (CNNs) in computer vision (CV), computer-aided automated methods have become the preferred methods for medical image diagnoses. Recently, Transformer has emerged in CV with performance comparable to CNN. Assisted diagnoses are often performed on resource-limited computing devices. The deployments of deep learning (DL) models are limited by the number of parameters and the computation. This study provides a DL training framework that introduces a model compression method of knowledge distillation (KD) in the classification of leukocytes, using small models instead of large ones, to achieve accurate results. Firstly, large models with CNN or Transformer structure are pre-trained on the mixed leukocyte dataset with 25, 830 original images. Then, the dark knowledge of the pre-trained large models is extracted by KD, and the small models are trained. Finally, the best performing small model is selected as the final prediction model, which achieves 98.31% testing accuracy on the mixed dataset. The proposed framework on the enhanced BCCD dataset achieves 99.88% testing accuracy, which is better than other methods. It effectively combines the advantages of large and small models to meet the requirements of low resource consumption and high accuracy.