Communication-efficient and Byzantine-robust distributed learning with statistical guarantee. (May 2023)

• Record Type: Journal Article
• Authors: Zhou, Xingcai; Chang, Le; Xu, Pengfei; Lv, Shaogao
• Is Part Of: Pattern recognition. Volume 137(2023)
• Publication Date: 2023-05
• Subjects: Distributed learning -- Byzantine failure -- Communication efficiency -- Surrogate likelihood -- Proximal algorithm; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2023.109312
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Both communication efficiency and robustness for convex distributed learning are taken into accounts simultaneously, which is very rare in existing related literatures. The current work develops two communication-efficient and robust distributed learning algorithms for convex problems. Particularly, the proposed algorithms are provably robust against Byzantine failures, and also achieve optimal statistical rates for strong convex losses and convex (non-smooth) penalties. For typical statistical models such as generalized linear models, our results show that statistical errors dominate optimization errors in finite iterations. Simulated and real data experiments are conducted to demonstrate the comparable performance of our algorithms. Abstract: Communication efficiency and robustness are two major issues in modern distributed learning frameworks. This is due to the practical situations where some computing nodes may have limited communication power or may behave adversarial behaviors. To address the two issues simultaneously, this paper develops two communication-efficient and robust distributed learning algorithms for convex problems. Our motivation is based on surrogate likelihood framework and the median and trimmed mean operations. Particularly, the proposed algorithms are provably robust against Byzantine failures, and also achieve optimal statistical rates for strong convex losses and convex (non-smooth) penalties. For typical statistical models such as generalized linear models, our results show that statistical errors dominate optimization errors in finite iterations. Simulated and real data experiments are conducted to demonstrate the numerical performance of our algorithms.