Composite quantile regression for massive datasets. Issue 5 (3rd September 2018)

• Record Type: Journal Article
• Authors: Jiang, Rong; Hu, Xueping; Yu, Keming; Qian, Weimin
• Is Part Of: Statistics. Volume 52:Issue 5(2018)
• Page Start: 980
• Page End: 1004
• Publication Date: 2018-09-03
• Subjects: Massive dataset -- divide and conquer -- composite quantile regression -- variable selection; Mathematical statistics -- Periodicals; 519.505
• DOI: 10.1080/02331888.2018.1500579
• Languages: English
• ISSNs: 0233-1888

Abstract

ABSTRACT: Analysis of massive datasets is challenging owing to limitations of computer primary memory. Composite quantile regression (CQR) is a robust and efficient estimation method. In this paper, we extend CQR to massive datasets and propose a divide-and-conquer CQR method. The basic idea is to split the entire dataset into several blocks, applying the CQR method for data in each block, and finally combining these regression results via weighted average. The proposed approach significantly reduces the required amount of primary memory, and the resulting estimate will be as efficient as if the entire data set is analysed simultaneously. Moreover, to improve the efficiency of CQR, we propose a weighted CQR estimation approach. To achieve sparsity with high-dimensional covariates, we develop a variable selection procedure to select significant parametric components and prove the method possessing the oracle property. Both simulations and data analysis are conducted to illustrate the finite sample performance of the proposed methods.