Context-guided entropy minimization for semi-supervised domain adaptation. (October 2022)

• Record Type: Journal Article
• Authors: Ma, Ning; Bu, Jiajun; Lu, Lixian; Wen, Jun; Zhou, Sheng; Zhang, Zhen; Gu, Jingjun; Li, Haifeng; Yan, Xifeng
• Is Part Of: Neural networks. Volume 154(2022)
• Page Start: 270
• Page End: 282
• Publication Date: 2022-10
• Subjects: Domain adaptation -- Transfer learning -- Semi-supervised learning; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2022.07.011
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: Semi-Supervised Domain Adaptation has been widely studied with various approaches to address domain shift with labeled source-domain data combined with scarcely labeled target-domain data. Model adaptation is becoming promising with a paradigm of source pre-training and target fine-tuning, which eliminates the simultaneous availability of data from both domains and makes for data privacy. Among the model adaptation methods, Entropy Minimization (EM) is popularly incorporated to encourage a low-density separation on target samples. However, EM tends to brutally force models to make over-confident predictions, which could make the models collapse with deteriorated performance. In this paper, we first study the over-confidence of EM with a quantitative analysis, which shows the importance of capturing the dependency among labels. To address this issue, we propose to guide EM via longitudinal self-distillation. Specifically, we produce a dynamic "teacher" label distribution during training by constructing a graph on target data and perform pseudo-label propagation to encourage the "teacher" distribution to capture context category dependency based on a global data structure. Then EM is guided longitudinally by distilling the learned label distribution to combat the brute-force over-confidence. Extensive experiments demonstrate the effectiveness of our methods.