Random Barzilai–Borwein step size for mini-batch algorithms. (June 2018)

• Record Type: Journal Article
• Authors: Yang, Zhuang; Wang, Cheng; Zhang, Zhemin; Li, Jonathan
• Is Part Of: Engineering applications of artificial intelligence. Volume 72(2018)
• Page Start: 124
• Page End: 135
• Publication Date: 2018-06
• Subjects: Stochastic gradient descent -- Mini batches -- Barzilai–Borwein method -- Variance reduction -- Convex optimization; Engineering -- Data processing -- Periodicals; Artificial intelligence -- Periodicals; Expert systems (Computer science) -- Periodicals; Ingénierie -- Informatique -- Périodiques; Intelligence artificielle -- Périodiques; Systèmes experts (Informatique) -- Périodiques; Artificial intelligence; Engineering -- Data processing; Expert systems (Computer science); Periodicals; 620.00285
• DOI: 10.1016/j.engappai.2018.03.017
• Languages: English
• ISSNs: 0952-1976

Abstract

Abstract: Mini-batch algorithms, a well-studied, highly popular approach in stochastic optimization methods, are used by practitioners because of their ability to accelerate training through better use of parallel processing power and reduction of stochastic variance. However, mini-batch algorithms often employ either a diminishing step size or a tuning step size by hand, which, in practice, can be time consuming. In this paper, we propose using the improved Barzilai–Borwein (BB) method to automatically compute step sizes for the state of the art mini-batch algorithm (mini-batch semi-stochastic gradient descent (mS2GD) method), which leads to a new algorithm: mS2GD-RBB. We theoretically prove that mS2GD-RBB converges with a linear convergence rate for strongly convex objective functions. To further validate the efficacy and scalability of the improved BB method, we introduce it into another modern mini-batch algorithm, Accelerated Mini-Batch Prox SVRG (Acc-Prox-SVRG) method. In a machine learning context, numerical experiments on three benchmark data sets indicate that the proposed methods outperform some advanced stochastic optimization methods.