SATS: Self-attention transfer for continual semantic segmentation. (June 2023)

• Record Type: Journal Article
• Authors: Qiu, Yiqiao; Shen, Yixing; Sun, Zhuohao; Zheng, Yanchong; Chang, Xiaobin; Zheng, Weishi; Wang, Ruixuan
• Is Part Of: Pattern recognition. Volume 138(2023)
• Publication Date: 2023-06
• Subjects: Continual learning -- Semantic segmentation -- Self-attention transfer -- Class-specific region pooling; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2023.109383
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Propose distilling self-attention maps for continual semantic segmentation. Propose a class-specific region pooling for relational knowledge transfer. First study to innovatively apply Transformer to continual semantic segmentation. Can flexibly combine the proposed method with existing strategies. Extensive evaluation on multiple benchmarks and settings with SOTA performance. Abstract: Continually learning to segment more and more types of image regions is a desired capability for many intelligent systems. However, such continual semantic segmentation exhibits catastrophic forgetting issues similar to those of continual classification learning. Unlike the existing knowledge distillation strategies for alleviating this problem, transferring a new type of information, namely, the relationships between elements (e.g., pixels) within each image that can capture both within-class and between-class knowledge, is proposed in this study. Such information can be effectively obtained from self-attention maps in a Transformer-style segmentation model. Considering that pixels belonging to the same class in each image typically share similar visual properties, a class-specific region pooling operator is novelly applied to provide reliable relationship information for knowledge transfer. Extensive evaluations on multiple public benchmarks reveal that the proposed self-attention transfer method can effectively alleviate the catastrophic forgetting issue. Furthermore, flexible combinations of the proposed method with widely adopted strategies considerably outperform state-of-the-art solutions.