Stochastic proximal linear method for structured non-convex problems. (2nd September 2020) - Hazan, Tamir; Sabach, Shoham; Voldman, Sergey
Abstract : In this work, motivated by the challenging task of learning a deep neural network, we consider optimization problems that consist of minimizing a finite-sum of non-convex and non-smooth functions, where the non-smoothness appears as the maximum of non-convex functions with Lipschitz continuous gradient. Due to the large size of the sum, in practice, we focus here on stochastic first-order methods and propose the Stochastic Proximal Linear Method (SPLM) that is based on minimizing an appropriate majorizer at each iteration and is guaranteed to almost surely converge to a critical point of the objective function, where we also proves its convergence rate in finding critical points.