Multi-scale vision transformer classification model with self-supervised learning and dilated convolution. (October 2022)

• Record Type: Journal Article
• Authors: Xing, Liping; Jin, Hongmei; Li, Hong-an; Li, Zhanli
• Is Part Of: Computers & electrical engineering. Volume 103(2022)
• Publication Date: 2022-10
• Subjects: Self-attention -- Vision transformer -- Multi-scale -- Self-supervised -- Depthwise separable dilated convolution -- Local features; Computer engineering -- Periodicals; Electrical engineering -- Periodicals; Electrical engineering -- Data processing -- Periodicals; Ordinateurs -- Conception et construction -- Périodiques; Électrotechnique -- Périodiques; Électrotechnique -- Informatique -- Périodiques; Computer engineering; Electrical engineering; Electrical engineering -- Data processing; Periodicals; Electronic journals; 621.302854
• DOI: 10.1016/j.compeleceng.2022.108270
• Languages: English
• ISSNs: 0045-7906

Abstract

Abstract: Benefiting from the advantages of good parallelism and features that support long-distance dependency modeling, a variety of ViT models based on the self-attention mechanism show outstanding performance in image classification tasks. However, these works have poor classification accuracy when trained on small datasets owing to insufficient attention toward local features. Therefore, this study presents a new self-supervised multi-scale ViT classification model, SMvT. This model adopts twin-tower architectures as the self-supervised framework and the hierarchical Swin Transformer as the backbone and proposes a MDP embedding layer to fully pay attention to local details. We investigated the model's performance pretrained using the lightweight ImageNet dataset. Compared with recent self-supervised Transformers for vision, such as MoCo v3 and DINO, the classification accuracy of the SMvT has improved by up to 15.2%. SMvT combines self-supervised learning and depthwise separable dilated convolution, which is a lightweight and high generalization ViT model supporting cross-scale attention modeling. Graphical abstract: Highlights: Constructed a self-supervised vision Transformer classification model. Proposed a multi-scale dilated pyramid (MDP) embedding layer. Designed a depthwise separable dilated convolution (DSD conv).