When sub-band features meet attention mechanism while knowledge distillation for sound classification. (30th June 2022)

• Record Type: Journal Article
• Authors: Tripathi, Achyut Mani; Paul, Konark
• Is Part Of: Applied acoustics. Volume 195(2022)
• Publication Date: 2022-06-30
• Subjects: Deep Networks -- Ensemble -- Knowledge Distillation -- Spectrogram -- Sound Classification; Acoustical engineering -- Periodicals; Periodicals; 620.2
• DOI: 10.1016/j.apacoust.2022.108813
• Languages: English
• ISSNs: 0003-682X

Abstract

Highlights: First contribution is the introduction of a framework to sub-divide the spectrogram into multiple sub-band spectrograms and explore use of three state-of-the-art attention mechanisms while training the teacher models to supervise the student model in order to focus on the spectrogram's relevant areas while neglecting the non-relevant regions. Second contribution is to create an ensemble of logits of multiple teacher models using three distinct ensemble techniques while distilling the knowledge to the student model. Finally, a comprehensive study that includes the explainable nature of the student model, the effects of multiple sub-bands, attention mechanisms and the selection of best ensemble technique for KD. Abstract: This paper introduces a novel knowledge distillation (KD) framework that distills the knowledge from multiple deep models trained on spectrogram features obtained by dividing the spectrogram into multiple sub-band spectrograms. The deep models learned from sub-band spectrograms prevent information loss while performing knowledge distillation from a teacher model to a student model receiving the full spectrogram as an input. The student models' performance is evaluated on three benchmark sound datasets, viz., the ESC-10, RAVDESS and Audio MNIST datasets. Impact of three state-of-the-art attention mechanisms is investigated thoroughly to enhance the final accuracy of the student model supervised by the proposed knowledge distillation framework for sound classification. Experiments and results show that the performance of the student model in the presence of state-of-the-art attention mechanisms is comparable and competitive to state-of-the-art techniques. Moreover, the student model trained on the Audio MNIST dataset attains an hitherto unpublished accuracy of 98.24%, a new benchmark for the Audio MNIST dataset. Additionally, Grad-CAM visualization of the spectrograms is provided to understand the spectrogram's relevant regions and explain why the model classifies a signal into a specific class. The code used in this work is available at: https://github.com/achyutmani/Sub-Band-Guided-KD-for-Sound-Classification .