Contrastive-weighted self-supervised model for long-tailed data classification with vision transformer augmented. (1st September 2022)

• Record Type: Journal Article
• Authors: Hou, Rujie; Chen, Jinglong; Feng, Yong; Liu, Shen; He, Shuilong; Zhou, Zitong
• Is Part Of: Mechanical systems and signal processing. Volume 177(2022)
• Publication Date: 2022-09-01
• Subjects: Long-tailed data -- Fault classification -- Contrastive self-supervised learning -- Vision transformer; Structural dynamics -- Periodicals; Vibration -- Periodicals; Constructions -- Dynamique -- Périodiques; Vibration -- Périodiques; Structural dynamics; Vibration; Periodicals; 621
• DOI: 10.1016/j.ymssp.2022.109174
• Languages: English
• ISSNs: 0888-3270

Abstract

Highlights: We proposed Contrastive-weighted Self-supervised Model (CSM) for fault classification under long-tailed data. We composited positive and negative samples for contrastive learning and transfer the encoder to downstream task for final classification. The imbalanced learning strategy is adopted in the pretext task for improving the training efficiency. The augmented vision transformer is adopted as the encoder to strengthen the representation learning ability of samples. Abstract: In practical mechanical fault diagnosis, it's difficult to obtain fault data and the acquisition between normal and fault data is in great imbalance, usually presenting a long-tail distribution state. When training the long-tailed data directly, the imbalanced label may cause label bias, leading to better performance on dominant classes but poorer generalization on tail classes. To address this problem, we adopt two-stage training and propose contrastive-weighted self-supervised model (CSM) with augmented vision transformer which merges the imbalanced learning strategies during the pre-training. In the pretext task, we abandon the label information and adopt contrastive learning by constructing positive and negative sample pairs of each sample. Training it approach to positive sample and away from negatives via augmented vision transformer. The imbalanced strategies are implemented by adaptively weighting to the similarity loss with effective number of samples to learn a better original representation of long-tailed data. In the downstream task, long-tailed data with label are used to fine-tune the pre-trained encoder, which can effectively achieve the final classification task. The experiments under two datasets demonstrate that the pre-training stage can effectively learn a good initialized encoder and can be used in the downstream tasks for better long-tailed data classification.