Efficient Perturbation Inference and Expandable Network for continual learning. (February 2023)

• Record Type: Journal Article
• Authors: Du, Fei; Yang, Yun; Zhao, Ziyuan; Zeng, Zeng
• Is Part Of: Neural networks. Volume 159(2023)
• Page Start: 97
• Page End: 106
• Publication Date: 2023-02
• Subjects: Continual learning -- Dynamic networks -- Class incremental learning -- Uncertainty inference; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2022.10.030
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: Although humans are capable of learning new tasks without forgetting previous ones, most neural networks fail to do so because learning new tasks could override the knowledge acquired from previous data. In this work, we alleviate this issue by proposing a novel Efficient Perturbation Inference and Expandable Network (EPIE-Net), which dynamically expands lightweight task-specific decoders for new classes and utilizes a mixed-label uncertainty strategy to improve the robustness. Moreover, we calculate the average probability of perturbed samples at inference, which can generally improve the performance of the model. Experimental results show that our method consistently outperforms other methods with fewer parameters in class incremental learning benchmarks. For example, on the CIFAR-100 10 steps setup, our method achieves an average accuracy of 76.33% and the last accuracy of 65.93% within only 3.46M average parameters.