Stochastic proximal linear method for structured non-convex problems. (2nd September 2020)

• Record Type: Journal Article
• Authors: Hazan, Tamir; Sabach, Shoham; Voldman, Sergey
• Is Part Of: Optimization methods and software. Volume 35:Number 5(2020)
• Page Start: 921
• Page End: 937
• Publication Date: 2020-09-02
• Subjects: Non-convex non-smooth minimization -- proximal-gradient algorithms -- stochastic -- convergence analysis; Primary: 90C25 -- Secondary: 26B25 -- 49M27 -- 52A41 -- 65K05; Mathematical optimization -- Periodicals; Algorithms -- Periodicals; 519.7
• DOI: 10.1080/10556788.2020.1754413
• Languages: English
• ISSNs: 1055-6788

Abstract

Abstract : In this work, motivated by the challenging task of learning a deep neural network, we consider optimization problems that consist of minimizing a finite-sum of non-convex and non-smooth functions, where the non-smoothness appears as the maximum of non-convex functions with Lipschitz continuous gradient. Due to the large size of the sum, in practice, we focus here on stochastic first-order methods and propose the Stochastic Proximal Linear Method (SPLM) that is based on minimizing an appropriate majorizer at each iteration and is guaranteed to almost surely converge to a critical point of the objective function, where we also proves its convergence rate in finding critical points.