Scatter matrix concordance as a diagnostic for regressions on subsets of data. (23rd September 2015)

• Record Type: Journal Article
• Authors: Kane, Michael J.; Lewis, Bryan; Tatikonda, Sekhar; Urbanek, Simon
• Is Part Of: Statistical analysis and data mining. Volume 9:Number 4(2016)
• Page Start: 249
• Page End: 259
• Publication Date: 2015-09-23
• Subjects: statistical concordance -- linear models -- correlation matrices; Data mining -- Statistical methods -- Periodicals; 006.312
• DOI: 10.1002/sam.11283
• Languages: English
• ISSNs: 1932-1864

Abstract

Abstract : Linear regression models depend directly on the design matrix and its properties. Techniques that efficiently estimate model coefficients by partitioning rows of the design matrix are increasingly becoming popular for large‐scale problems because they fit well with modern parallel computing architectures. We propose a simple measure of concordance between a design matrix and a subset of its rows that estimates how well a subset captures the variance‐covariance structure of a larger data set. We illustrate the use of this measure in a heuristic method for selecting row partition sizes that balance statistical and computational efficiency goals in real‐world problems.