Two tales of variable selection for high dimensional regression: Screening and model building. (23rd April 2014)

• Record Type: Journal Article
• Authors: Liu, Cong; Shi, Tao; Lee, Yoonkyung
• Is Part Of: Statistical analysis and data mining. Volume 7:Number 2(2014)
• Page Start: 140
• Page End: 159
• Publication Date: 2014-04-23
• Subjects: Data mining -- Statistical methods -- Periodicals; 006.312
• DOI: 10.1002/sam.11219
• Languages: English
• ISSNs: 1932-1864

Abstract

<abstract abstract-type="main" xml:lang="en"> <title>Abstract</title> <p>Variable selection plays an important role in high‐dimensional regression problems where a large number of variables are given as potential predictors of a response of interest. Typically, it arises at two stages of statistical modeling, namely screening and formal model building, with different goals. Screening aims at filtering out <italic>irrelevant variables</italic> prior to model building where a formal description of a functional relation between the variables screened for relevance and the response is sought. Accordingly, proper comparison of variable selection methods calls for evaluation criteria that reflect the differential goals: accuracy in ranking order of variables for screening and prediction accuracy for formal modeling.</p> <p>Without delineating the difference in the two aspects, confounding comparisons of various screening and selection methods have often been made in the literature, which may lead to misleading conclusions. In this paper, we present comprehensive numerical studies for comparison of four commonly used screening and selection procedures: correlation screening (also known as sure independence screening), forward selection, LASSO and SCAD. By clearly differentiating screening and model building, we highlight the situations where the performance of these procedures might differ. In addition, we propose a new method for cross‐validation for LASSO. Furthermore, we discuss connections to relevant comparison studies that appeared in the recent literature to clarify different findings and conclusions.</p> </abstract>