DPCL: Contrastive representation learning with differential privacy. Issue 11 (26th August 2022)

• Record Type: Journal Article
• Authors: Li, Wenjun; Yan, Anli; Wu, Di; Zhu, Taoyu; Huang, Teng; Luo, Xuandi; Wang, Shaowei
• Is Part Of: International journal of intelligent systems. Volume 37:Issue 11(2022)
• Page Start: 9701
• Page End: 9725
• Publication Date: 2022-08-26
• Subjects: contrastive learning -- differential privacy -- privacy protection -- private training -- sensitivity analysis; Artificial intelligence -- Periodicals; Expert systems (Computer science) -- Periodicals; Intelligence artificielle -- Périodiques; Systèmes experts (Informatique) -- Périodiques; 006.3
• DOI: 10.1002/int.23002
• Languages: English
• ISSNs: 0884-8173

Abstract

Abstract: With the proliferation of unlabeled data, increasing efforts have been devoted to unsupervised learning. As one of the most representative branches of unsupervised learning, contrastive learning has made great progress with its high efficiency. Unfortunately, privacy threats to contrastive learning have become sophisticated, making it imperative to develop effective technologies that can deal with such threats. To alleviate the privacy issue in contrastive learning, we propose some novel techniques based on differential privacy, which aim at reducing the high sensitivity of gradient in the private training caused by interactive contrastive learning. Specifically, we add differentially private protection to the connection point related to different per‐example gradients, which decreases the sensitivity of the gradients significantly. Our experiments on SimCLR and the Barlow Twins show that our approach is superior since it is more accurate while maintaining the same level of privacy protection.