Using the area under an estimated ROC curve to test the adequacy of binary predictors*. Issue 1 (2nd January 2019)

• Record Type: Journal Article
• Authors: Lieli, Robert P.; Hsu, Yu-Chin
• Is Part Of: Journal of nonparametric statistics. Volume 31:Issue 1(2019)
• Page Start: 100
• Page End: 130
• Publication Date: 2019-01-02
• Subjects: Area under the ROC curve -- overfitting -- in-sample hypothesis testing -- binary classification -- model evaluation; Nonparametric statistics -- Periodicals; 519.5
• DOI: 10.1080/10485252.2018.1537440
• Languages: English
• ISSNs: 1048-5252

Abstract

ABSTRACT: We consider using the area under an empirical receiver operating characteristic curve to test the hypothesis that a predictive index combined with a range of cutoffs performs no better than pure chance in forecasting a binary outcome. This corresponds to the null hypothesis that the area in question, denoted as AUC, is 1/2. We show that if the predictive index comes from a first-stage regression model estimated over the same data set, then testing the null based on the standard asymptotic normality results leads to severe size distortion in general settings. We then analytically derive the proper asymptotic null distribution of the empirical AUC in a special case; namely, when the first-stage regressors are Bernoulli random variables. This distribution can be utilised to construct a fully in-sample test ofH 0 : AUC = 1 / 2 with correct size and more power than out-of-sample tests based on sample splitting, though practical application becomes cumbersome with more than two regressors.