Variable Screening for Sparse Online Regression. Issue 1 (2nd January 2023)

• Record Type: Journal Article
• Authors: Liang, Jingwei; Poon, Clarice
• Is Part Of: Journal of computational and graphical statistics. Volume 32:Issue 1(2023)
• Page Start: 275
• Page End: 293
• Publication Date: 2023-01-02
• Subjects: Finite activity identification -- Nonsmooth regularization -- Screening rules -- Sparsity promoting regularization -- Stochastic gradient descent; Mathematical statistics -- Data processing -- Periodicals; Mathematical statistics -- Graphic methods -- Periodicals; 519.50285
• DOI: 10.1080/10618600.2022.2099872
• Languages: English
• ISSNs: 1061-8600

Abstract

Abstract: Sparsity-promoting regularizers are widely used to impose low-complexity structure (e.g., l 1 -norm for sparsity) to the regression coefficients of supervised learning. In the realm of deterministic optimization, the sequence generated by iterative algorithms (such as proximal gradient descent) exhibit "finite activity identification" property, that is, they can identify the low-complexity structure of the solution in a finite number of iterations. However, many online algorithms (such as proximal stochastic gradient descent) do not have this property owing to the vanishing step-size and nonvanishing variance. In this article, by combining with a screening rule, we show how to eliminate useless features of the iterates generated by online algorithms, and thereby enforce finite sparsity identification. One advantage of our scheme is that when combined with any convergent online algorithm, sparsity properties imposed by the regularizer can be exploited to improve computational efficiency. Numerically, significant acceleration can be obtained.