Improving the detection of unusual observations in high‐dimensional settings. (8th December 2017)

• Record Type: Journal Article
• Authors: Ullah, Insha; Pawley, Matthew D.M.; Smith, Adam N.H.; Jones, Beatrix
• Is Part Of: Australian & New Zealand journal of statistics. Volume 59:Number 4(2017)
• Page Start: 449
• Page End: 462
• Publication Date: 2017-12-08
• Subjects: high‐dimensional data -- outlier detection -- shrinkage estimator; Statistics -- Periodicals; 519.5
• DOI: 10.1111/anzs.12210
• Languages: English
• ISSNs: 1369-1473

Abstract

Summary: Multivariate control charts are used to monitor stochastic processes for changes and unusual observations. Hotelling's T 2 statistic is calculated for each new observation and an out‐of‐control signal is issued if it goes beyond the control limits. However, this classical approach becomes unreliable as the number of variables p approaches the number of observations n, and impossible when p exceeds n . In this paper, we devise an improvement to the monitoring procedure in high‐dimensional settings. We regularise the covariance matrix to estimate the baseline parameter and incorporate a leave‐one‐out re‐sampling approach to estimate the empirical distribution of future observations. An extensive simulation study demonstrates that the new method outperforms the classical Hotelling T 2 approach in power, and maintains appropriate false positive rates. We demonstrate the utility of the method using a set of quality control samples collected to monitor a gas chromatography–mass spectrometry apparatus over a period of 67 days. Abstract : Using a James‐Stein shrinkage estimator for the covariance extends the Hotelling procedure to the case where dimension exceeds sample size.